[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260924 00:20:24.668036 812 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.203.62:36271
I20260924 00:20:24.669610 812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:24.670737 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:24.693499 821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:24.693547 819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:24.694510 818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:24.695876 812 server_base.cc:1034] running on GCE node
I20260924 00:20:25.917356 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:25.917593 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:25.917735 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209225917722 us; error 0 us; skew 500 ppm
I20260924 00:20:25.918510 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:25.925486 812 webserver.cc:460] Webserver started at http://127.0.203.62:38461/ using document root <none> and password file <none>
I20260924 00:20:25.926376 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:25.926590 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:25.927013 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:25.934252 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/master-0-root/instance:
uuid: "1daf06ac7fd8419f841222befa8e5d3d"
format_stamp: "Formatted at 2026-09-24 00:20:25 on dist-test-slave-r18p"
I20260924 00:20:25.942905 812 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260924 00:20:25.948312 827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:25.949607 812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 00:20:25.949904 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/master-0-root
uuid: "1daf06ac7fd8419f841222befa8e5d3d"
format_stamp: "Formatted at 2026-09-24 00:20:25 on dist-test-slave-r18p"
I20260924 00:20:25.950214 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:25.998840 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:26.000404 812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:26.000874 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:26.079821 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36271
I20260924 00:20:26.079916 878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36271 every 8 connection(s)
I20260924 00:20:26.087169 879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:26.111440 879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d: Bootstrap starting.
I20260924 00:20:26.117311 879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:26.118985 879 log.cc:824] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:26.123716 879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d: No bootstrap required, opened a new log
I20260924 00:20:26.147567 879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1daf06ac7fd8419f841222befa8e5d3d" member_type: VOTER }
I20260924 00:20:26.148113 879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:26.148307 879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daf06ac7fd8419f841222befa8e5d3d, State: Initialized, Role: FOLLOWER
I20260924 00:20:26.149145 879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1daf06ac7fd8419f841222befa8e5d3d" member_type: VOTER }
I20260924 00:20:26.149605 879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:20:26.149842 879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:20:26.150117 879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:26.153888 879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1daf06ac7fd8419f841222befa8e5d3d" member_type: VOTER }
I20260924 00:20:26.154515 879 leader_election.cc:302] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1daf06ac7fd8419f841222befa8e5d3d; no voters:
I20260924 00:20:26.156459 879 leader_election.cc:288] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:20:26.156842 882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:26.159785 882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 1 LEADER]: Becoming Leader. State: Replica: 1daf06ac7fd8419f841222befa8e5d3d, State: Running, Role: LEADER
I20260924 00:20:26.161104 882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1daf06ac7fd8419f841222befa8e5d3d" member_type: VOTER }
I20260924 00:20:26.161978 879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:20:26.173487 883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1daf06ac7fd8419f841222befa8e5d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1daf06ac7fd8419f841222befa8e5d3d" member_type: VOTER } }
I20260924 00:20:26.173431 884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1daf06ac7fd8419f841222befa8e5d3d. Latest consensus state: current_term: 1 leader_uuid: "1daf06ac7fd8419f841222befa8e5d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1daf06ac7fd8419f841222befa8e5d3d" member_type: VOTER } }
I20260924 00:20:26.174156 883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:26.174285 884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:26.178125 890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:20:26.191699 890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:20:26.195355 812 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260924 00:20:26.210600 890 catalog_manager.cc:1347] Generated new cluster ID: bb4c71ace5f44d6f87935399a16f9fe9
I20260924 00:20:26.210918 890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:20:26.261971 890 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:20:26.264072 890 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:20:26.276710 890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d: Generated new TSK 0
I20260924 00:20:26.277881 890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:20:26.327435 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:26.338605 901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:26.348954 902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:26.349900 812 server_base.cc:1034] running on GCE node
W20260924 00:20:26.349855 904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:26.351039 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:26.351306 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:26.351500 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209226351484 us; error 0 us; skew 500 ppm
I20260924 00:20:26.352293 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:26.356684 812 webserver.cc:460] Webserver started at http://127.0.203.1:32821/ using document root <none> and password file <none>
I20260924 00:20:26.357385 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:26.357632 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:26.357963 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:26.359688 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-0-root/instance:
uuid: "ded90ff3c146407a822cc07a683e33b9"
format_stamp: "Formatted at 2026-09-24 00:20:26 on dist-test-slave-r18p"
I20260924 00:20:26.367605 812 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s
I20260924 00:20:26.373075 909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.374001 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:20:26.374437 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-0-root
uuid: "ded90ff3c146407a822cc07a683e33b9"
format_stamp: "Formatted at 2026-09-24 00:20:26 on dist-test-slave-r18p"
I20260924 00:20:26.374817 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:26.417670 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:26.419344 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:26.422082 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:26.427891 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:26.428221 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.428521 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:26.428742 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.527849 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:38173
I20260924 00:20:26.527937 971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:38173 every 8 connection(s)
I20260924 00:20:26.535825 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:26.546391 976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:26.550626 977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:26.559904 812 server_base.cc:1034] running on GCE node
W20260924 00:20:26.560994 979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:26.562178 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:26.562448 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:26.562747 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209226562726 us; error 0 us; skew 500 ppm
I20260924 00:20:26.563516 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:26.567112 972 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36271
I20260924 00:20:26.567106 812 webserver.cc:460] Webserver started at http://127.0.203.2:42021/ using document root <none> and password file <none>
I20260924 00:20:26.567625 972 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:26.567899 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:26.568171 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:26.568486 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:26.568583 972 heartbeater.cc:511] Master 127.0.203.62:36271 requested a full tablet report, sending...
I20260924 00:20:26.569994 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-1-root/instance:
uuid: "b01fcc3af0f740529463824ade56fcc7"
format_stamp: "Formatted at 2026-09-24 00:20:26 on dist-test-slave-r18p"
I20260924 00:20:26.571723 844 ts_manager.cc:194] Registered new tserver with Master: ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173)
I20260924 00:20:26.574021 844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59678
I20260924 00:20:26.577853 812 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260924 00:20:26.582557 984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.583905 812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260924 00:20:26.584192 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-1-root
uuid: "b01fcc3af0f740529463824ade56fcc7"
format_stamp: "Formatted at 2026-09-24 00:20:26 on dist-test-slave-r18p"
I20260924 00:20:26.584458 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:26.618599 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:26.619834 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:26.623066 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:26.625190 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:26.625380 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.625665 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:26.625797 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.732748 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:46547
I20260924 00:20:26.732852 1046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:46547 every 8 connection(s)
I20260924 00:20:26.739645 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:26.748379 1050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:26.751035 1051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:26.760800 1047 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36271
I20260924 00:20:26.761261 1047 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:26.762279 1047 heartbeater.cc:511] Master 127.0.203.62:36271 requested a full tablet report, sending...
W20260924 00:20:26.763453 1053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:26.764426 812 server_base.cc:1034] running on GCE node
I20260924 00:20:26.764591 844 ts_manager.cc:194] Registered new tserver with Master: b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:26.765417 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:26.765650 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:26.765826 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209226765812 us; error 0 us; skew 500 ppm
I20260924 00:20:26.766350 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:26.766717 844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59686
I20260924 00:20:26.769851 812 webserver.cc:460] Webserver started at http://127.0.203.3:36169/ using document root <none> and password file <none>
I20260924 00:20:26.770480 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:26.770728 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:26.771044 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:26.772320 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-2-root/instance:
uuid: "1b4bbe0422db401db0a37c388d720c01"
format_stamp: "Formatted at 2026-09-24 00:20:26 on dist-test-slave-r18p"
I20260924 00:20:26.778584 812 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 00:20:26.782398 1058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.783288 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:20:26.783619 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-2-root
uuid: "1b4bbe0422db401db0a37c388d720c01"
format_stamp: "Formatted at 2026-09-24 00:20:26 on dist-test-slave-r18p"
I20260924 00:20:26.783977 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790209224507772-812-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:26.803337 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:26.804879 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:26.806770 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:26.810062 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:26.810323 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.810590 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:26.810794 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:26.916185 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:41697
I20260924 00:20:26.916241 1120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:41697 every 8 connection(s)
I20260924 00:20:26.936482 1121 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36271
I20260924 00:20:26.936883 1121 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:26.937682 1121 heartbeater.cc:511] Master 127.0.203.62:36271 requested a full tablet report, sending...
I20260924 00:20:26.939601 844 ts_manager.cc:194] Registered new tserver with Master: 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:26.940188 812 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017819046s
I20260924 00:20:26.941706 844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59702
I20260924 00:20:26.946503 812 test_util.cc:268] Using random seed: -1861503063
I20260924 00:20:27.007513 844 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59704:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:20:27.011745 844 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:20:27.140843 1085 tablet_service.cc:1467] Processing CreateTablet for tablet c20b6a7dd2ed4f9da1c679eadf285605 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260924 00:20:27.142554 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20b6a7dd2ed4f9da1c679eadf285605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.143888 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec4554cc9314cec92eb5b7c40797696 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260924 00:20:27.142124 1086 tablet_service.cc:1467] Processing CreateTablet for tablet e3883c040d834e2da77d51c1b30cc0c6 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260924 00:20:27.150864 1084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec4554cc9314cec92eb5b7c40797696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.158856 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 9976c883f10e45ae95ee12dda05212b1 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260924 00:20:27.161597 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3883c040d834e2da77d51c1b30cc0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.162967 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7bb05c9df540c09ccff1dce38119ae (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260924 00:20:27.158586 1080 tablet_service.cc:1467] Processing CreateTablet for tablet b9a9dc94f8b84b6ab7e76e7aea7f27e2 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260924 00:20:27.174304 1081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7bb05c9df540c09ccff1dce38119ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.175103 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec4554cc9314cec92eb5b7c40797696 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260924 00:20:27.169461 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 26273297bebf41048141ce34242ae0d5 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260924 00:20:27.168135 1078 tablet_service.cc:1467] Processing CreateTablet for tablet aa24aa2041174acb84bbec10cdeb698c (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260924 00:20:27.183504 1083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9976c883f10e45ae95ee12dda05212b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.189000 935 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec4554cc9314cec92eb5b7c40797696 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260924 00:20:27.190411 935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec4554cc9314cec92eb5b7c40797696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.178843 1008 tablet_service.cc:1467] Processing CreateTablet for tablet 26273297bebf41048141ce34242ae0d5 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260924 00:20:27.175602 1079 tablet_service.cc:1467] Processing CreateTablet for tablet 18cac6776aec4050ac6fbd87e3c3701e (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260924 00:20:27.185161 937 tablet_service.cc:1467] Processing CreateTablet for tablet e3883c040d834e2da77d51c1b30cc0c6 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260924 00:20:27.196981 1010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec4554cc9314cec92eb5b7c40797696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.184233 1007 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7bb05c9df540c09ccff1dce38119ae (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260924 00:20:27.193391 1009 tablet_service.cc:1467] Processing CreateTablet for tablet 9976c883f10e45ae95ee12dda05212b1 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260924 00:20:27.182660 936 tablet_service.cc:1467] Processing CreateTablet for tablet c20b6a7dd2ed4f9da1c679eadf285605 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260924 00:20:27.182339 1012 tablet_service.cc:1467] Processing CreateTablet for tablet e3883c040d834e2da77d51c1b30cc0c6 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260924 00:20:27.179067 1006 tablet_service.cc:1467] Processing CreateTablet for tablet b9a9dc94f8b84b6ab7e76e7aea7f27e2 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260924 00:20:27.201360 1078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa24aa2041174acb84bbec10cdeb698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.201960 933 tablet_service.cc:1467] Processing CreateTablet for tablet 26273297bebf41048141ce34242ae0d5 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260924 00:20:27.199608 1004 tablet_service.cc:1467] Processing CreateTablet for tablet aa24aa2041174acb84bbec10cdeb698c (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260924 00:20:27.200425 1011 tablet_service.cc:1467] Processing CreateTablet for tablet c20b6a7dd2ed4f9da1c679eadf285605 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260924 00:20:27.195550 1005 tablet_service.cc:1467] Processing CreateTablet for tablet 18cac6776aec4050ac6fbd87e3c3701e (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260924 00:20:27.212355 931 tablet_service.cc:1467] Processing CreateTablet for tablet b9a9dc94f8b84b6ab7e76e7aea7f27e2 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
W20260924 00:20:27.217572 892 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:20:27.215780 934 tablet_service.cc:1467] Processing CreateTablet for tablet 9976c883f10e45ae95ee12dda05212b1 (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260924 00:20:27.215127 1008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26273297bebf41048141ce34242ae0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.219771 937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3883c040d834e2da77d51c1b30cc0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.205838 932 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7bb05c9df540c09ccff1dce38119ae (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260924 00:20:27.212352 1080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a9dc94f8b84b6ab7e76e7aea7f27e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.207629 930 tablet_service.cc:1467] Processing CreateTablet for tablet 18cac6776aec4050ac6fbd87e3c3701e (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260924 00:20:27.219732 929 tablet_service.cc:1467] Processing CreateTablet for tablet aa24aa2041174acb84bbec10cdeb698c (DEFAULT_TABLE table=test-workload [id=89526563204b48aba7363990fd1a6942]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260924 00:20:27.227771 1009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9976c883f10e45ae95ee12dda05212b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.242133 931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a9dc94f8b84b6ab7e76e7aea7f27e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.242283 1079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cac6776aec4050ac6fbd87e3c3701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.257268 1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a9dc94f8b84b6ab7e76e7aea7f27e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.259961 1011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20b6a7dd2ed4f9da1c679eadf285605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.267148 1012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3883c040d834e2da77d51c1b30cc0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.262670 932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7bb05c9df540c09ccff1dce38119ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.268888 1082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26273297bebf41048141ce34242ae0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.269390 930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cac6776aec4050ac6fbd87e3c3701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.271441 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa24aa2041174acb84bbec10cdeb698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.270526 933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26273297bebf41048141ce34242ae0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.273723 1005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cac6776aec4050ac6fbd87e3c3701e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.282522 1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7bb05c9df540c09ccff1dce38119ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.291947 929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa24aa2041174acb84bbec10cdeb698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.307293 934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9976c883f10e45ae95ee12dda05212b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.374711 1140 tablet_bootstrap.cc:492] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:27.378206 936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20b6a7dd2ed4f9da1c679eadf285605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:27.430377 1139 tablet_bootstrap.cc:492] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:27.434466 1141 tablet_bootstrap.cc:492] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:27.503731 1141 tablet_bootstrap.cc:654] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.505025 1140 tablet_bootstrap.cc:654] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.560166 1139 tablet_bootstrap.cc:654] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.574079 1141 tablet_bootstrap.cc:492] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:27.574833 1141 ts_tablet_manager.cc:1397] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.015s
I20260924 00:20:27.577957 1141 raft_consensus.cc:348] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.578783 1141 raft_consensus.cc:374] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.579115 972 heartbeater.cc:503] Master 127.0.203.62:36271 was elected leader, sending a full tablet report...
I20260924 00:20:27.579097 1141 raft_consensus.cc:729] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.581414 1141 consensus_queue.cc:260] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.583066 1140 tablet_bootstrap.cc:492] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:27.583607 1121 heartbeater.cc:503] Master 127.0.203.62:36271 was elected leader, sending a full tablet report...
I20260924 00:20:27.585562 1140 ts_tablet_manager.cc:1397] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.211s user 0.013s sys 0.018s
I20260924 00:20:27.588153 1139 tablet_bootstrap.cc:492] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:27.589149 1139 ts_tablet_manager.cc:1397] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.191s user 0.010s sys 0.007s
I20260924 00:20:27.592062 1139 raft_consensus.cc:348] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.592798 1139 raft_consensus.cc:374] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.593067 1139 raft_consensus.cc:729] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.594646 1141 ts_tablet_manager.cc:1428] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.019s user 0.008s sys 0.002s
I20260924 00:20:27.595132 1140 raft_consensus.cc:348] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.595907 1140 raft_consensus.cc:374] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.596174 1140 raft_consensus.cc:729] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.595583 1141 tablet_bootstrap.cc:492] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:27.596801 1140 consensus_queue.cc:260] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.594178 1139 consensus_queue.cc:260] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.603796 1141 tablet_bootstrap.cc:654] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.606288 1047 heartbeater.cc:503] Master 127.0.203.62:36271 was elected leader, sending a full tablet report...
I20260924 00:20:27.606793 1140 ts_tablet_manager.cc:1428] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.021s user 0.005s sys 0.004s
I20260924 00:20:27.608506 1140 tablet_bootstrap.cc:492] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:27.624276 1140 tablet_bootstrap.cc:654] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.636942 1139 ts_tablet_manager.cc:1428] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.047s user 0.019s sys 0.001s
I20260924 00:20:27.640949 1139 tablet_bootstrap.cc:492] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:27.648030 1139 tablet_bootstrap.cc:654] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.657269 1141 tablet_bootstrap.cc:492] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:27.657747 1141 ts_tablet_manager.cc:1397] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.062s user 0.021s sys 0.031s
I20260924 00:20:27.658661 1140 tablet_bootstrap.cc:492] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:27.659132 1140 ts_tablet_manager.cc:1397] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.006s
I20260924 00:20:27.660764 1141 raft_consensus.cc:348] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.661551 1141 raft_consensus.cc:374] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.661983 1141 raft_consensus.cc:729] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.661571 1140 raft_consensus.cc:348] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.662501 1140 raft_consensus.cc:374] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.662906 1140 raft_consensus.cc:729] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.662714 1141 consensus_queue.cc:260] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.663667 1140 consensus_queue.cc:260] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.665124 1141 ts_tablet_manager.cc:1428] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:20:27.665766 1140 ts_tablet_manager.cc:1428] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260924 00:20:27.666042 1141 tablet_bootstrap.cc:492] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:27.667224 1140 tablet_bootstrap.cc:492] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:27.672456 1141 tablet_bootstrap.cc:654] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.675268 1140 tablet_bootstrap.cc:654] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
W20260924 00:20:27.675278 1122 tablet.cc:2369] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:20:27.684405 1141 tablet_bootstrap.cc:492] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:27.685338 1141 ts_tablet_manager.cc:1397] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.013s
I20260924 00:20:27.688005 1141 raft_consensus.cc:348] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.688769 1141 raft_consensus.cc:374] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.689076 1141 raft_consensus.cc:729] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.689924 1141 consensus_queue.cc:260] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.691438 1140 tablet_bootstrap.cc:492] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:27.691912 1140 ts_tablet_manager.cc:1397] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20260924 00:20:27.692742 1141 ts_tablet_manager.cc:1428] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260924 00:20:27.693714 1141 tablet_bootstrap.cc:492] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:27.694475 1140 raft_consensus.cc:348] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.695212 1140 raft_consensus.cc:374] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.695564 1140 raft_consensus.cc:729] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.696460 1146 raft_consensus.cc:482] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:27.696728 1140 consensus_queue.cc:260] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.696907 1146 raft_consensus.cc:504] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.700345 1146 leader_election.cc:288] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:27.700536 1141 tablet_bootstrap.cc:654] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.718339 1139 tablet_bootstrap.cc:492] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:27.718868 1139 ts_tablet_manager.cc:1397] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.008s
I20260924 00:20:27.740984 1145 raft_consensus.cc:482] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:27.741660 1145 raft_consensus.cc:504] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.753800 1139 raft_consensus.cc:348] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.754801 1139 raft_consensus.cc:374] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.755149 1139 raft_consensus.cc:729] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.756932 1139 consensus_queue.cc:260] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.770587 1140 ts_tablet_manager.cc:1428] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.078s user 0.012s sys 0.011s
I20260924 00:20:27.774672 1140 tablet_bootstrap.cc:492] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:27.770696 1145 leader_election.cc:288] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:27.781294 1139 ts_tablet_manager.cc:1428] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.062s user 0.007s sys 0.003s
I20260924 00:20:27.782104 1141 tablet_bootstrap.cc:492] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:27.784950 1139 tablet_bootstrap.cc:492] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:27.797312 1141 ts_tablet_manager.cc:1397] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.104s user 0.027s sys 0.025s
I20260924 00:20:27.798905 1139 tablet_bootstrap.cc:654] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.803395 1141 raft_consensus.cc:348] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.806743 1141 raft_consensus.cc:374] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.807916 1141 raft_consensus.cc:729] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.816231 1141 consensus_queue.cc:260] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.816967 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:27.820454 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:27.821313 1096 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:27.822053 1140 tablet_bootstrap.cc:654] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.823331 986 leader_election.cc:302] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:27.818332 947 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:27.824601 1153 raft_consensus.cc:2781] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:27.829876 1153 raft_consensus.cc:482] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:27.834550 1153 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:27.829912 1141 ts_tablet_manager.cc:1428] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.032s user 0.003s sys 0.004s
I20260924 00:20:27.841272 1146 raft_consensus.cc:482] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:27.841723 1146 raft_consensus.cc:504] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.842171 1141 tablet_bootstrap.cc:492] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:27.843603 1146 leader_election.cc:288] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697), ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173)
I20260924 00:20:27.844749 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:27.845566 947 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:27.847272 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:27.848045 988 leader_election.cc:302] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:27.848994 1146 raft_consensus.cc:2781] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:27.849339 1146 raft_consensus.cc:482] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:27.849637 1146 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:20:27.849598 986 leader_election.cc:341] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:27.864331 1146 raft_consensus.cc:504] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.866966 1146 leader_election.cc:288] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Requested vote from peers 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697), ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173)
I20260924 00:20:27.867686 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:27.868458 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
W20260924 00:20:27.868963 986 leader_election.cc:341] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:27.869023 947 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:27.875293 1139 tablet_bootstrap.cc:492] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:27.875976 1139 ts_tablet_manager.cc:1397] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.094s user 0.010s sys 0.021s
I20260924 00:20:27.879341 1139 raft_consensus.cc:348] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.879805 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bb05c9df540c09ccff1dce38119ae" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:27.880254 1139 raft_consensus.cc:374] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.880548 1139 raft_consensus.cc:729] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.881222 1139 consensus_queue.cc:260] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
W20260924 00:20:27.881695 1062 leader_election.cc:341] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:27.883433 1139 ts_tablet_manager.cc:1428] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260924 00:20:27.886091 1139 tablet_bootstrap.cc:492] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:27.892488 1139 tablet_bootstrap.cc:654] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.895507 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bb05c9df540c09ccff1dce38119ae" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:27.875321 1141 tablet_bootstrap.cc:654] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
W20260924 00:20:27.897372 1062 leader_election.cc:341] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:27.897051 1140 tablet_bootstrap.cc:492] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:27.897842 1062 leader_election.cc:302] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01; no voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:27.898666 1145 raft_consensus.cc:2726] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:27.898211 1140 ts_tablet_manager.cc:1397] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.008s
I20260924 00:20:27.902537 1145 raft_consensus.cc:482] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:27.902977 1145 raft_consensus.cc:504] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.904927 1139 tablet_bootstrap.cc:492] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:27.905392 1139 ts_tablet_manager.cc:1397] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.014s
I20260924 00:20:27.905489 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:27.904841 1145 leader_election.cc:288] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:27.906174 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
W20260924 00:20:27.907665 1062 leader_election.cc:341] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:20:27.908576 1062 leader_election.cc:341] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:27.909220 1062 leader_election.cc:302] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01; no voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:27.908911 1139 raft_consensus.cc:348] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.909974 1139 raft_consensus.cc:374] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.910154 1145 raft_consensus.cc:2726] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:27.910270 1139 raft_consensus.cc:729] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.910961 1139 consensus_queue.cc:260] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.913548 1139 ts_tablet_manager.cc:1428] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260924 00:20:27.914861 1139 tablet_bootstrap.cc:492] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:27.922772 1139 tablet_bootstrap.cc:654] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.925498 1147 raft_consensus.cc:482] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:27.926134 1147 raft_consensus.cc:504] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.928571 1147 leader_election.cc:288] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:27.929915 1146 raft_consensus.cc:482] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:27.930411 1146 raft_consensus.cc:504] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.931952 1146 leader_election.cc:288] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:27.932629 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:27.932919 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:27.933337 946 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
W20260924 00:20:27.934122 986 leader_election.cc:341] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:27.934755 988 leader_election.cc:302] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters: 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:27.901324 1140 raft_consensus.cc:348] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.935575 1146 raft_consensus.cc:2781] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:27.938481 1146 raft_consensus.cc:482] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:27.938928 1146 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:27.939791 1140 raft_consensus.cc:374] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.940173 1140 raft_consensus.cc:729] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.941042 1140 consensus_queue.cc:260] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.942377 1147 raft_consensus.cc:482] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:27.942902 1147 raft_consensus.cc:504] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.945283 1147 leader_election.cc:288] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:27.949888 1141 tablet_bootstrap.cc:492] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:27.950510 1141 ts_tablet_manager.cc:1397] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.109s user 0.012s sys 0.001s
I20260924 00:20:27.954836 1141 raft_consensus.cc:348] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.955880 1141 raft_consensus.cc:374] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:27.956238 1141 raft_consensus.cc:729] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:27.957051 1141 consensus_queue.cc:260] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.959024 1153 raft_consensus.cc:504] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:27.960322 947 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:27.961110 1153 leader_election.cc:288] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:27.962746 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:27.963340 1096 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:27.963717 988 leader_election.cc:302] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters: 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:27.965390 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:27.966056 947 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:27.967765 1153 raft_consensus.cc:2781] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:27.968245 1153 raft_consensus.cc:686] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:27.969250 1140 ts_tablet_manager.cc:1428] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.070s user 0.011s sys 0.002s
I20260924 00:20:27.969022 1153 consensus_queue.cc:237] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:27.970435 1140 tablet_bootstrap.cc:492] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:27.972359 1141 ts_tablet_manager.cc:1428] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.021s user 0.006s sys 0.003s
I20260924 00:20:27.973858 1141 tablet_bootstrap.cc:492] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:27.982385 1139 tablet_bootstrap.cc:492] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:27.982890 1139 ts_tablet_manager.cc:1397] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.068s user 0.020s sys 0.011s
I20260924 00:20:27.983026 1141 tablet_bootstrap.cc:654] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:27.989523 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:27.991786 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
W20260924 00:20:27.993461 911 leader_election.cc:341] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:20:27.994187 911 leader_election.cc:341] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:27.996690 1141 tablet_bootstrap.cc:492] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:27.997536 1141 ts_tablet_manager.cc:1397] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.005s
I20260924 00:20:27.995703 841 catalog_manager.cc:5495] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:27.998523 1139 raft_consensus.cc:348] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.001334 1139 raft_consensus.cc:374] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.001693 1139 raft_consensus.cc:729] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.002846 1139 consensus_queue.cc:260] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.003969 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:28.005034 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:28.005496 1139 ts_tablet_manager.cc:1428] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
W20260924 00:20:28.007252 913 leader_election.cc:341] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:28.007658 1139 tablet_bootstrap.cc:492] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:28.007707 913 leader_election.cc:302] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ded90ff3c146407a822cc07a683e33b9; no voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:28.007897 1141 raft_consensus.cc:348] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.009042 1147 raft_consensus.cc:2726] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:28.009125 1141 raft_consensus.cc:374] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.009537 1141 raft_consensus.cc:729] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.010728 1141 consensus_queue.cc:260] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.011596 1140 tablet_bootstrap.cc:654] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.015920 1141 ts_tablet_manager.cc:1428] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260924 00:20:28.017490 1141 tablet_bootstrap.cc:492] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:28.021232 1139 tablet_bootstrap.cc:654] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.029349 1141 tablet_bootstrap.cc:654] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.032039 1147 raft_consensus.cc:482] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.033195 1147 raft_consensus.cc:504] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.036231 1147 leader_election.cc:288] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.037158 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:28.037160 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:28.037956 1021 raft_consensus.cc:2454] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 0.
I20260924 00:20:28.042706 1094 raft_consensus.cc:2454] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 0.
I20260924 00:20:28.044170 913 leader_election.cc:302] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.046062 1147 raft_consensus.cc:2781] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:28.046408 1147 raft_consensus.cc:482] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:28.046731 1147 raft_consensus.cc:3037] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.054160 1140 tablet_bootstrap.cc:492] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:28.056763 1140 ts_tablet_manager.cc:1397] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.087s user 0.015s sys 0.007s
I20260924 00:20:28.060352 1139 tablet_bootstrap.cc:492] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:28.060396 1141 tablet_bootstrap.cc:492] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:28.061201 1141 ts_tablet_manager.cc:1397] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.011s
I20260924 00:20:28.061172 1139 ts_tablet_manager.cc:1397] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.007s
I20260924 00:20:28.065649 947 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.065711 1146 raft_consensus.cc:504] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.067770 988 leader_election.cc:302] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.068228 1146 leader_election.cc:288] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.066232 1096 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.069418 1170 raft_consensus.cc:2781] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.070756 1170 raft_consensus.cc:686] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:28.071602 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:28.071754 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:28.073745 1022 raft_consensus.cc:2379] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded90ff3c146407a822cc07a683e33b9 in current term 1: Already voted for candidate b01fcc3af0f740529463824ade56fcc7 in this term.
I20260924 00:20:28.073444 947 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.072047 1170 consensus_queue.cc:237] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
W20260924 00:20:28.075142 986 leader_election.cc:341] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:28.066175 1140 raft_consensus.cc:348] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.077668 1140 raft_consensus.cc:374] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.077293 1141 raft_consensus.cc:348] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.077972 1140 raft_consensus.cc:729] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.078064 1141 raft_consensus.cc:374] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.078426 1141 raft_consensus.cc:729] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.078648 1140 consensus_queue.cc:260] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.079128 1141 consensus_queue.cc:260] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.081738 1141 ts_tablet_manager.cc:1428] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260924 00:20:28.082170 1140 ts_tablet_manager.cc:1428] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20260924 00:20:28.082683 1141 tablet_bootstrap.cc:492] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:28.083160 1140 tablet_bootstrap.cc:492] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:28.097313 1139 raft_consensus.cc:348] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.098296 1139 raft_consensus.cc:374] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.098636 1139 raft_consensus.cc:729] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.099275 1139 consensus_queue.cc:260] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.100248 1140 tablet_bootstrap.cc:654] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.101606 913 leader_election.cc:302] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ded90ff3c146407a822cc07a683e33b9; no voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:28.118315 1139 ts_tablet_manager.cc:1428] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.056s user 0.001s sys 0.018s
I20260924 00:20:28.119313 1139 tablet_bootstrap.cc:492] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:28.124507 1140 tablet_bootstrap.cc:492] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:28.125183 1140 ts_tablet_manager.cc:1397] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.009s
I20260924 00:20:28.127729 1141 tablet_bootstrap.cc:654] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.125118 843 catalog_manager.cc:5495] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:28.130949 1140 raft_consensus.cc:348] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.131793 1140 raft_consensus.cc:374] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.133325 1140 raft_consensus.cc:729] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.133728 1139 tablet_bootstrap.cc:654] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.134094 1140 consensus_queue.cc:260] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.136865 1140 ts_tablet_manager.cc:1428] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260924 00:20:28.137781 1181 raft_consensus.cc:482] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.138744 1140 tablet_bootstrap.cc:492] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:28.140417 1181 raft_consensus.cc:504] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.144189 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
W20260924 00:20:28.145622 913 leader_election.cc:341] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:28.148188 1141 tablet_bootstrap.cc:492] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:28.148648 1141 ts_tablet_manager.cc:1397] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.066s user 0.014s sys 0.006s
I20260924 00:20:28.151333 1140 tablet_bootstrap.cc:654] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.151216 1141 raft_consensus.cc:348] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.151932 1141 raft_consensus.cc:374] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.152216 1141 raft_consensus.cc:729] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.152842 1141 consensus_queue.cc:260] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.155251 1141 ts_tablet_manager.cc:1428] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 00:20:28.156497 1181 leader_election.cc:288] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.158416 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
W20260924 00:20:28.160271 911 leader_election.cc:341] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:28.161116 911 leader_election.cc:302] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ded90ff3c146407a822cc07a683e33b9; no voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:28.163620 1141 tablet_bootstrap.cc:492] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01: Bootstrap starting.
I20260924 00:20:28.162955 1181 raft_consensus.cc:2726] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:28.165563 1140 tablet_bootstrap.cc:492] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:28.167105 947 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.167057 1147 raft_consensus.cc:504] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.168073 1180 raft_consensus.cc:2726] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:28.166018 1140 ts_tablet_manager.cc:1397] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.002s
I20260924 00:20:28.169677 1147 leader_election.cc:288] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.166790 1139 tablet_bootstrap.cc:492] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:28.170500 1139 ts_tablet_manager.cc:1397] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.009s
I20260924 00:20:28.170795 988 leader_election.cc:302] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters: 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:28.171988 1170 raft_consensus.cc:2781] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.172427 1170 raft_consensus.cc:686] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:28.172487 1140 raft_consensus.cc:348] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.173329 1140 raft_consensus.cc:374] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.173664 1140 raft_consensus.cc:729] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.173894 1139 raft_consensus.cc:348] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.172398 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:28.174588 1139 raft_consensus.cc:374] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.174381 1140 consensus_queue.cc:260] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.174918 1096 raft_consensus.cc:3037] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.174908 1139 raft_consensus.cc:729] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.176072 1139 consensus_queue.cc:260] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.176802 1140 ts_tablet_manager.cc:1428] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:20:28.177776 1140 tablet_bootstrap.cc:492] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:28.178956 1139 ts_tablet_manager.cc:1428] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260924 00:20:28.179818 1139 tablet_bootstrap.cc:492] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:28.172397 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:28.181784 1022 raft_consensus.cc:3037] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.183490 1141 tablet_bootstrap.cc:654] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.186550 1139 tablet_bootstrap.cc:654] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.173329 1170 consensus_queue.cc:237] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.197314 1140 tablet_bootstrap.cc:654] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.203084 841 catalog_manager.cc:5495] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:28.220479 1145 raft_consensus.cc:482] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.221577 1145 raft_consensus.cc:504] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
W20260924 00:20:28.221977 892 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:20:28.223539 1145 leader_election.cc:288] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:28.225350 1139 tablet_bootstrap.cc:492] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:28.225091 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:28.225783 1139 ts_tablet_manager.cc:1397] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.006s
I20260924 00:20:28.225631 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
W20260924 00:20:28.226459 1062 leader_election.cc:341] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:28.226491 1021 raft_consensus.cc:2454] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 0.
I20260924 00:20:28.228155 1062 leader_election.cc:302] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters: ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.228921 1145 raft_consensus.cc:2781] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:28.228672 1139 raft_consensus.cc:348] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.229295 1145 raft_consensus.cc:482] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:28.229450 1139 raft_consensus.cc:374] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.229682 1145 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.229774 1139 raft_consensus.cc:729] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.231271 1141 tablet_bootstrap.cc:492] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01: No bootstrap required, opened a new log
I20260924 00:20:28.231673 1141 ts_tablet_manager.cc:1397] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.014s
I20260924 00:20:28.233413 1141 raft_consensus.cc:348] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.231271 1139 consensus_queue.cc:260] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.233908 1141 raft_consensus.cc:374] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.234206 1141 raft_consensus.cc:729] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.234881 1141 consensus_queue.cc:260] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.236444 1140 tablet_bootstrap.cc:492] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:28.237011 1140 ts_tablet_manager.cc:1397] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.009s
I20260924 00:20:28.237979 1139 ts_tablet_manager.cc:1428] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260924 00:20:28.238796 1139 tablet_bootstrap.cc:492] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9: Bootstrap starting.
I20260924 00:20:28.240516 1140 raft_consensus.cc:348] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.241662 1140 raft_consensus.cc:374] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.242067 1140 raft_consensus.cc:729] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.242913 1140 consensus_queue.cc:260] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.254791 1139 tablet_bootstrap.cc:654] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.257273 1140 ts_tablet_manager.cc:1428] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.020s user 0.005s sys 0.003s
I20260924 00:20:28.261729 1180 raft_consensus.cc:482] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.262244 1180 raft_consensus.cc:504] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.264410 1180 leader_election.cc:288] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.265988 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:28.267089 1021 raft_consensus.cc:2454] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 0.
I20260924 00:20:28.267980 1096 raft_consensus.cc:2454] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 1.
I20260924 00:20:28.268760 1140 tablet_bootstrap.cc:492] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7: Bootstrap starting.
I20260924 00:20:28.268841 1022 raft_consensus.cc:2454] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 1.
I20260924 00:20:28.271759 1190 raft_consensus.cc:482] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.274154 1141 ts_tablet_manager.cc:1428] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01: Time spent starting tablet: real 0.042s user 0.018s sys 0.007s
I20260924 00:20:28.274380 913 leader_election.cc:302] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.276948 913 leader_election.cc:302] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.278180 1147 raft_consensus.cc:2781] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:28.278360 1180 raft_consensus.cc:2781] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.276542 1190 raft_consensus.cc:504] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.278573 1147 raft_consensus.cc:482] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:28.279007 1147 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.279618 1180 raft_consensus.cc:686] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:28.279794 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:28.280577 1180 consensus_queue.cc:237] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.283104 1190 leader_election.cc:288] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:28.284605 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:28.284750 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:28.285256 947 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 0.
I20260924 00:20:28.285524 1140 tablet_bootstrap.cc:654] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:28.288060 1147 raft_consensus.cc:504] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.292730 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:28.293370 1022 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.299487 1140 tablet_bootstrap.cc:492] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7: No bootstrap required, opened a new log
I20260924 00:20:28.288369 1145 raft_consensus.cc:504] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.300354 1096 raft_consensus.cc:2379] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded90ff3c146407a822cc07a683e33b9 in current term 1: Already voted for candidate 1b4bbe0422db401db0a37c388d720c01 in this term.
I20260924 00:20:28.289646 1062 leader_election.cc:302] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.299907 1140 ts_tablet_manager.cc:1397] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.000s
I20260924 00:20:28.301760 1190 raft_consensus.cc:2781] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:28.302124 1190 raft_consensus.cc:482] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:20:28.301904 1062 leader_election.cc:341] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:28.302531 1190 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.302187 1145 leader_election.cc:288] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:28.305060 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:28.305872 947 raft_consensus.cc:2379] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b4bbe0422db401db0a37c388d720c01 in current term 1: Already voted for candidate ded90ff3c146407a822cc07a683e33b9 in this term.
I20260924 00:20:28.303103 1022 raft_consensus.cc:2454] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 1.
I20260924 00:20:28.307479 913 leader_election.cc:302] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.308262 1181 raft_consensus.cc:2781] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.310118 1190 raft_consensus.cc:504] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.312489 1190 leader_election.cc:288] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:28.315382 1140 raft_consensus.cc:348] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.316213 1140 raft_consensus.cc:374] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.316494 1140 raft_consensus.cc:729] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.317114 1140 consensus_queue.cc:260] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.318195 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:28.318971 1021 raft_consensus.cc:2379] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b4bbe0422db401db0a37c388d720c01 in current term 1: Already voted for candidate ded90ff3c146407a822cc07a683e33b9 in this term.
I20260924 00:20:28.319607 1140 ts_tablet_manager.cc:1428] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7: Time spent starting tablet: real 0.018s user 0.001s sys 0.004s
I20260924 00:20:28.320694 1147 leader_election.cc:288] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 1 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.322206 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:28.322597 841 catalog_manager.cc:5495] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:28.322214 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:28.324672 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:28.325002 1021 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.325428 1096 raft_consensus.cc:2379] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ded90ff3c146407a822cc07a683e33b9 in current term 1: Already voted for candidate 1b4bbe0422db401db0a37c388d720c01 in this term.
I20260924 00:20:28.328207 1062 leader_election.cc:302] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01; no voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.323236 947 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.333359 1181 raft_consensus.cc:686] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:28.334414 1181 consensus_queue.cc:237] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.335546 1139 tablet_bootstrap.cc:492] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9: No bootstrap required, opened a new log
I20260924 00:20:28.335966 1139 ts_tablet_manager.cc:1397] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9: Time spent bootstrapping tablet: real 0.097s user 0.016s sys 0.000s
I20260924 00:20:28.336629 1190 raft_consensus.cc:2726] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 00:20:28.337344 1021 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 1.
I20260924 00:20:28.338327 1139 raft_consensus.cc:348] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.338970 1139 raft_consensus.cc:374] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:28.339221 1139 raft_consensus.cc:729] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Initialized, Role: FOLLOWER
I20260924 00:20:28.339769 1139 consensus_queue.cc:260] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.340231 1062 leader_election.cc:302] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:28.343146 1190 raft_consensus.cc:2781] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.343034 947 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 1.
I20260924 00:20:28.343654 1190 raft_consensus.cc:686] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 LEADER]: Becoming Leader. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:28.344285 1190 consensus_queue.cc:237] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.346309 1139 ts_tablet_manager.cc:1428] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260924 00:20:28.351810 841 catalog_manager.cc:5495] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 1 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:28.359266 1190 raft_consensus.cc:482] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.359094 843 catalog_manager.cc:5495] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "1b4bbe0422db401db0a37c388d720c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: HEALTHY } } }
I20260924 00:20:28.359724 1190 raft_consensus.cc:504] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.361575 1190 leader_election.cc:288] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:28.363121 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:28.364121 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:28.364845 947 raft_consensus.cc:2379] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b4bbe0422db401db0a37c388d720c01 in current term 1: Already voted for candidate b01fcc3af0f740529463824ade56fcc7 in this term.
I20260924 00:20:28.365688 1062 leader_election.cc:302] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01; no voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.370093 1190 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.376657 1190 raft_consensus.cc:2726] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:28.417860 1190 raft_consensus.cc:482] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.418406 1190 raft_consensus.cc:504] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:28.420994 1190 leader_election.cc:288] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173)
I20260924 00:20:28.421815 1021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fcc3af0f740529463824ade56fcc7" is_pre_election: true
I20260924 00:20:28.422086 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:28.422820 947 raft_consensus.cc:2379] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b4bbe0422db401db0a37c388d720c01 in current term 1: Already voted for candidate b01fcc3af0f740529463824ade56fcc7 in this term.
I20260924 00:20:28.423645 1062 leader_election.cc:302] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01; no voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.424373 1190 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.428635 1190 raft_consensus.cc:2726] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:28.437440 1170 raft_consensus.cc:482] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.437964 1170 raft_consensus.cc:504] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.439937 1170 leader_election.cc:288] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.440822 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:28.440999 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:28.441663 947 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:28.441778 1096 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:28.442694 988 leader_election.cc:302] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.443905 1170 raft_consensus.cc:2781] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:28.444238 1170 raft_consensus.cc:482] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:28.444540 1170 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.451227 1170 raft_consensus.cc:504] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.454320 1170 leader_election.cc:288] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.455366 1096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:28.455348 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:28.456055 1096 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.456110 947 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.462124 1096 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.462363 1146 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.463106 947 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.463316 986 leader_election.cc:302] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:28.464247 1170 raft_consensus.cc:2781] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.464725 1170 raft_consensus.cc:686] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:28.465612 1170 consensus_queue.cc:237] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.482342 842 catalog_manager.cc:5495] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:28.493325 1153 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.537353 1153 raft_consensus.cc:482] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.537796 1153 raft_consensus.cc:504] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.539358 1153 leader_election.cc:288] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.540267 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:28.540439 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:28.540807 947 raft_consensus.cc:2454] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:28.541090 1094 raft_consensus.cc:2454] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:28.541788 988 leader_election.cc:302] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.542531 1153 raft_consensus.cc:2781] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:28.542876 1153 raft_consensus.cc:482] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:28.543162 1153 raft_consensus.cc:3037] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.548480 1153 raft_consensus.cc:504] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.550514 1153 leader_election.cc:288] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.551285 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:28.551609 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:28.551929 947 raft_consensus.cc:3037] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.552129 1094 raft_consensus.cc:3037] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.558135 1094 raft_consensus.cc:2454] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.558135 947 raft_consensus.cc:2454] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.559576 986 leader_election.cc:302] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:28.560372 1153 raft_consensus.cc:2781] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.560763 1153 raft_consensus.cc:686] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:28.561543 1153 consensus_queue.cc:237] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.569877 843 catalog_manager.cc:5495] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:28.576686 1153 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.595643 1153 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.633431 1153 raft_consensus.cc:482] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:28.633891 1153 raft_consensus.cc:504] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.635957 1153 leader_election.cc:288] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.636791 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bb05c9df540c09ccff1dce38119ae" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9" is_pre_election: true
I20260924 00:20:28.637012 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bb05c9df540c09ccff1dce38119ae" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01" is_pre_election: true
I20260924 00:20:28.637494 947 raft_consensus.cc:2454] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:28.637589 1094 raft_consensus.cc:2454] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 0.
I20260924 00:20:28.638545 986 leader_election.cc:302] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:28.639178 1153 raft_consensus.cc:2781] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:28.639523 1153 raft_consensus.cc:482] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:28.639792 1153 raft_consensus.cc:3037] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.644287 1153 raft_consensus.cc:504] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.645757 1153 leader_election.cc:288] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:28.647106 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bb05c9df540c09ccff1dce38119ae" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:28.647253 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7bb05c9df540c09ccff1dce38119ae" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:28.647631 947 raft_consensus.cc:3037] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.647881 1094 raft_consensus.cc:3037] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:28.651948 947 raft_consensus.cc:2454] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.652885 988 leader_election.cc:302] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:28.653551 1153 raft_consensus.cc:2781] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:28.653884 1094 raft_consensus.cc:2454] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 1.
I20260924 00:20:28.654467 1153 raft_consensus.cc:686] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:28.655196 1153 consensus_queue.cc:237] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:28.662632 842 catalog_manager.cc:5495] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:28.676720 1153 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.689538 1170 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.807310 1181 consensus_queue.cc:1035] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:28.818149 1195 consensus_queue.cc:1035] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.819535 1195 consensus_queue.cc:1035] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.832336 1190 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:28.855268 1195 consensus_queue.cc:1035] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.873173 1190 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:28.918578 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:39206
I20260924 00:20:28.919304 1020 raft_consensus.cc:595] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Received request to transfer leadership to TS ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.924572 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:39206
I20260924 00:20:28.925177 1020 raft_consensus.cc:595] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Received request to transfer leadership to TS ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.930011 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:39206
I20260924 00:20:28.930601 1020 raft_consensus.cc:595] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Received request to transfer leadership to TS ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.947530 1096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
mode: GRACEFUL
new_leader_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:35962
I20260924 00:20:28.948195 1096 raft_consensus.cc:595] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 LEADER]: Received request to transfer leadership to TS ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:28.963335 947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26273297bebf41048141ce34242ae0d5"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
mode: GRACEFUL
new_leader_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:48940
I20260924 00:20:28.963871 947 raft_consensus.cc:595] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Received request to transfer leadership to TS b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:28.974741 947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
mode: GRACEFUL
new_leader_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:48940
I20260924 00:20:28.975332 947 raft_consensus.cc:595] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Received request to transfer leadership to TS b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:28.979909 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:39206
I20260924 00:20:28.980455 1020 raft_consensus.cc:595] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Received request to transfer leadership to TS 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:28.984812 1170 consensus_queue.cc:1035] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.995430 1206 raft_consensus.cc:979] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7: : Instructing follower 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:28.995543 1146 consensus_queue.cc:1035] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:28.995924 1206 raft_consensus.cc:1067] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Signalling peer 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:28.999876 1153 consensus_queue.cc:1035] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:29.001972 1094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:35944
I20260924 00:20:29.002507 1094 raft_consensus.cc:482] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:29.002835 1094 raft_consensus.cc:3037] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.014678 1094 raft_consensus.cc:504] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.015859 1096 raft_consensus.cc:1226] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Rejecting Update request from peer b01fcc3af0f740529463824ade56fcc7 for earlier term 1. Current term is 2. Ops: []
I20260924 00:20:29.018543 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:29.019163 1020 raft_consensus.cc:3032] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:29.019474 1020 raft_consensus.cc:729] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:29.020213 1020 consensus_queue.cc:260] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.021555 1020 raft_consensus.cc:3037] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.027968 1094 leader_election.cc:288] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 2 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:29.030771 1020 raft_consensus.cc:2454] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 2.
I20260924 00:20:29.032958 1062 leader_election.cc:302] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:29.035115 1153 consensus_queue.cc:1035] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:29.035769 1218 raft_consensus.cc:2781] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:29.036286 1218 raft_consensus.cc:686] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 2 LEADER]: Becoming Leader. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:29.032230 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa24aa2041174acb84bbec10cdeb698c" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:29.036942 947 raft_consensus.cc:3037] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.037159 1218 consensus_queue.cc:237] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.048218 842 catalog_manager.cc:5495] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 reported cstate change: term changed from 1 to 2, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "1b4bbe0422db401db0a37c388d720c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: HEALTHY } } }
I20260924 00:20:29.054241 947 raft_consensus.cc:2454] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 2.
I20260924 00:20:29.057379 1170 raft_consensus.cc:979] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.057821 1170 raft_consensus.cc:1067] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Signalling peer ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.059711 946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:48922
I20260924 00:20:29.060359 946 raft_consensus.cc:482] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:29.060808 946 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.066962 946 raft_consensus.cc:504] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.070150 946 leader_election.cc:288] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:29.070106 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:29.070899 1020 raft_consensus.cc:3032] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:29.070808 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:29.071316 1020 raft_consensus.cc:729] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:29.071471 1094 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.072110 1020 consensus_queue.cc:260] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.073371 1020 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.078205 1094 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.079269 911 leader_election.cc:302] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:29.080086 1195 raft_consensus.cc:2781] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:29.080523 1195 raft_consensus.cc:686] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:29.080452 1020 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.081429 1195 consensus_queue.cc:237] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.088680 842 catalog_manager.cc:5495] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 1 to 2, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 2 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:29.113806 1153 raft_consensus.cc:979] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.114328 1170 raft_consensus.cc:1067] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Signalling peer ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.115952 946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:48922
I20260924 00:20:29.116983 946 raft_consensus.cc:482] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:29.117450 946 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.122663 946 raft_consensus.cc:504] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:29.124342 946 leader_election.cc:288] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Requested vote from peers 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:29.125272 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:29.125386 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:29.125962 1094 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.126047 1020 raft_consensus.cc:3032] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:29.126468 1020 raft_consensus.cc:729] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:29.127291 1020 consensus_queue.cc:260] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:29.128597 1020 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.132902 1094 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.134120 911 leader_election.cc:302] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:29.135097 1195 raft_consensus.cc:2781] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:29.135532 1020 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.135684 1195 raft_consensus.cc:686] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:29.136641 1195 consensus_queue.cc:237] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:29.144580 842 catalog_manager.cc:5495] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 1 to 2, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 2 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } }
I20260924 00:20:29.190924 1170 consensus_queue.cc:1035] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:29.201674 1146 consensus_queue.cc:1035] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:29.222874 1146 raft_consensus.cc:979] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.223302 1146 raft_consensus.cc:1067] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Signalling peer ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.224689 946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:48922
I20260924 00:20:29.225250 946 raft_consensus.cc:482] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:29.225593 946 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.230170 946 raft_consensus.cc:504] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.231901 946 leader_election.cc:288] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:29.233273 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:29.233812 1020 raft_consensus.cc:3032] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:29.234109 1020 raft_consensus.cc:729] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:29.234812 1020 consensus_queue.cc:260] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.235888 1020 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.240644 1181 raft_consensus.cc:979] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9: : Instructing follower b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:29.241079 1181 raft_consensus.cc:1067] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Signalling peer b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:29.241331 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:29.241942 1094 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.242866 1022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:39196
I20260924 00:20:29.243386 1022 raft_consensus.cc:482] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:29.243745 1022 raft_consensus.cc:3037] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.246896 1094 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.248699 911 leader_election.cc:302] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:29.249881 1195 raft_consensus.cc:2781] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:29.250382 1195 raft_consensus.cc:686] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:29.251223 1195 consensus_queue.cc:237] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.252382 1020 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.253603 1022 raft_consensus.cc:504] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.256807 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:29.256575 1022 leader_election.cc:288] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 2 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:29.257417 946 raft_consensus.cc:3032] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:29.257774 946 raft_consensus.cc:729] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:29.259135 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cac6776aec4050ac6fbd87e3c3701e" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:29.259697 1094 raft_consensus.cc:3037] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.259295 946 consensus_queue.cc:260] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.260749 946 raft_consensus.cc:3037] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.264139 841 catalog_manager.cc:5495] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 1 to 2, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 2 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:29.268054 1094 raft_consensus.cc:2454] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 2.
I20260924 00:20:29.268054 946 raft_consensus.cc:2454] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 2.
I20260924 00:20:29.269465 988 leader_election.cc:302] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:29.270422 1146 raft_consensus.cc:2781] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:29.270957 1146 raft_consensus.cc:686] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 2 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:29.271847 1146 consensus_queue.cc:237] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.279106 841 catalog_manager.cc:5495] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 1 to 2, leader changed from ded90ff3c146407a822cc07a683e33b9 (127.0.203.1) to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 2 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:29.357946 1195 raft_consensus.cc:979] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9: : Instructing follower b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:29.358443 1147 raft_consensus.cc:1067] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Signalling peer b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:29.359829 1022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26273297bebf41048141ce34242ae0d5"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:39196
I20260924 00:20:29.360376 1022 raft_consensus.cc:482] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:29.360733 1022 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.365468 1022 raft_consensus.cc:504] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.367072 1022 leader_election.cc:288] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 2 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:29.368105 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:29.368083 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:29.368719 1094 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.368788 946 raft_consensus.cc:3032] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:29.369194 946 raft_consensus.cc:729] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:29.370048 946 consensus_queue.cc:260] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.371048 946 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.374519 1217 raft_consensus.cc:979] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01: : Instructing follower ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.374928 1218 raft_consensus.cc:1067] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 LEADER]: Signalling peer ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:29.376362 947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:48924
I20260924 00:20:29.376891 947 raft_consensus.cc:482] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:29.377116 1094 raft_consensus.cc:2454] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 2.
I20260924 00:20:29.377279 947 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.377737 946 raft_consensus.cc:2454] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 2.
I20260924 00:20:29.378199 986 leader_election.cc:302] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:29.379135 1146 raft_consensus.cc:2781] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:29.379595 1146 raft_consensus.cc:686] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:29.380452 1146 consensus_queue.cc:237] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.384232 947 raft_consensus.cc:504] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.386413 947 leader_election.cc:288] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:29.386945 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:29.387563 1022 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.387904 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:29.387583 841 catalog_manager.cc:5495] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 1 to 2, leader changed from ded90ff3c146407a822cc07a683e33b9 (127.0.203.1) to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 2 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:29.388435 1094 raft_consensus.cc:3032] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:29.388773 1094 raft_consensus.cc:729] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:29.389508 1094 consensus_queue.cc:260] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.392172 1094 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:29.392982 1022 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.394158 913 leader_election.cc:302] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:29.395043 1147 raft_consensus.cc:2781] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:29.395479 1147 raft_consensus.cc:686] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:29.396243 1147 consensus_queue.cc:237] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:29.398005 1094 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 2.
I20260924 00:20:29.403019 841 catalog_manager.cc:5495] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 1 to 2, leader changed from 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3) to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 2 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:29.469565 1022 raft_consensus.cc:1261] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.470739 1218 consensus_queue.cc:1035] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.478024 947 raft_consensus.cc:1261] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.479610 1218 consensus_queue.cc:1035] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.530750 1094 raft_consensus.cc:1261] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.531997 1147 consensus_queue.cc:1035] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.541275 1022 raft_consensus.cc:1261] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.543066 1181 consensus_queue.cc:1035] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:20:29.598942 1095 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.600273 1230 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.608129 1022 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.609838 1230 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.706859 947 raft_consensus.cc:1261] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.708003 1146 consensus_queue.cc:1035] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.716939 1095 raft_consensus.cc:1261] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.718189 1170 consensus_queue.cc:1035] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.765381 947 raft_consensus.cc:1261] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.766577 1170 consensus_queue.cc:1035] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.774016 1095 raft_consensus.cc:1261] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.776412 1146 consensus_queue.cc:1035] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.814854 1095 raft_consensus.cc:1261] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.816036 1230 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.823593 1022 raft_consensus.cc:1261] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.825033 1230 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.881253 1095 raft_consensus.cc:1261] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.882412 1230 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:29.890070 1022 raft_consensus.cc:1261] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:29.891517 1230 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:32.007872 812 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260924 00:20:32.008622 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:32.040753 1022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:39220
I20260924 00:20:32.041392 1022 raft_consensus.cc:595] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Received request to transfer leadership to TS 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:32.053475 947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
mode: GRACEFUL
new_leader_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:48942
I20260924 00:20:32.054010 947 raft_consensus.cc:595] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Received request to transfer leadership to TS 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:32.054973 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260924 00:20:32.055531 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:32.224792 1146 raft_consensus.cc:979] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:32.225265 1243 raft_consensus.cc:1067] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Signalling peer 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:32.226701 1094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:35944
I20260924 00:20:32.227166 1094 raft_consensus.cc:482] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:32.227450 1094 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:32.232558 1094 raft_consensus.cc:504] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:32.234099 1094 leader_election.cc:288] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 2 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:32.234979 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:32.235195 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:32.235447 947 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:32.235783 1022 raft_consensus.cc:3032] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:32.236058 1022 raft_consensus.cc:729] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:32.236665 1022 consensus_queue.cc:260] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:32.237692 1022 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:32.239785 947 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 2.
I20260924 00:20:32.240782 1062 leader_election.cc:302] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:32.241546 1239 raft_consensus.cc:2781] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:32.242199 1022 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 2.
I20260924 00:20:32.242642 1239 raft_consensus.cc:686] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 LEADER]: Becoming Leader. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:32.243450 1239 consensus_queue.cc:237] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:32.249714 843 catalog_manager.cc:5495] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 reported cstate change: term changed from 1 to 2, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "1b4bbe0422db401db0a37c388d720c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: HEALTHY } } }
I20260924 00:20:32.299391 1195 raft_consensus.cc:979] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9: : Instructing follower 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:32.299842 1195 raft_consensus.cc:1067] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Signalling peer 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:32.301290 1094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:35960
I20260924 00:20:32.301756 1094 raft_consensus.cc:482] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:32.302042 1094 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:32.306181 1094 raft_consensus.cc:504] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:32.307542 1094 leader_election.cc:288] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 3 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173)
I20260924 00:20:32.308444 1022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:32.308696 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:32.309026 1022 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:32.309257 947 raft_consensus.cc:3032] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:20:32.309599 947 raft_consensus.cc:729] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:32.310349 947 consensus_queue.cc:260] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:32.311517 947 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:32.315532 1022 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 3.
I20260924 00:20:32.316566 1062 leader_election.cc:302] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:32.317299 1239 raft_consensus.cc:2781] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:20:32.317299 947 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 3.
I20260924 00:20:32.317916 1239 raft_consensus.cc:686] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 LEADER]: Becoming Leader. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:32.318725 1239 consensus_queue.cc:237] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:32.324805 843 catalog_manager.cc:5495] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 reported cstate change: term changed from 2 to 3, leader changed from ded90ff3c146407a822cc07a683e33b9 (127.0.203.1) to 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "1b4bbe0422db401db0a37c388d720c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:32.675662 1022 raft_consensus.cc:1261] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:32.676921 1239 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:32.686818 947 raft_consensus.cc:1261] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:32.688445 1244 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:32.749020 1022 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:32.750166 1244 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:20:32.758673 947 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:32.760249 1239 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:20:33.056370 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:33.061209 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:33.061586 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:34.062474 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:34.067315 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:34.067692 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:35.068701 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:35.074388 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:35.074797 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:36.075570 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:36.080608 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:36.081027 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:37.081754 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:37.086898 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:37.087224 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:38.087908 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:38.092396 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:38.092707 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:39.093461 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:39.098099 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:39.098443 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:40.099117 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:40.104233 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:40.104547 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:41.105268 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:41.109865 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:41.110287 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:42.135895 812 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260924 00:20:42.139977 1094 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
mode: GRACEFUL
new_leader_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:35962
I20260924 00:20:42.140664 1094 raft_consensus.cc:595] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 LEADER]: Received request to transfer leadership to TS b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:42.145208 946 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
mode: GRACEFUL
new_leader_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:48940
I20260924 00:20:42.145658 946 raft_consensus.cc:595] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Received request to transfer leadership to TS b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:42.150255 946 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
mode: GRACEFUL
new_leader_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:48940
I20260924 00:20:42.150709 946 raft_consensus.cc:595] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Received request to transfer leadership to TS b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:42.154956 946 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
mode: GRACEFUL
new_leader_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:48940
I20260924 00:20:42.155458 946 raft_consensus.cc:595] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Received request to transfer leadership to TS b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:42.164809 1094 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
mode: GRACEFUL
new_leader_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:35962
I20260924 00:20:42.165284 1094 raft_consensus.cc:595] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 LEADER]: Received request to transfer leadership to TS b01fcc3af0f740529463824ade56fcc7
I20260924 00:20:42.269685 1266 raft_consensus.cc:979] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9: : Instructing follower b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.270112 1266 raft_consensus.cc:1067] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Signalling peer b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.271963 1022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:39196
I20260924 00:20:42.272477 1022 raft_consensus.cc:482] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:42.272792 1022 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.277943 1022 raft_consensus.cc:504] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.279683 1022 leader_election.cc:288] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:42.280718 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:42.281036 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:42.281531 1094 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.281726 946 raft_consensus.cc:3032] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:20:42.282119 946 raft_consensus.cc:729] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:42.282797 946 consensus_queue.cc:260] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.283788 946 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.286465 1094 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.287549 986 leader_election.cc:302] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:42.288321 1263 raft_consensus.cc:2781] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:20:42.289283 1263 raft_consensus.cc:686] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:42.289288 946 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.290272 1263 consensus_queue.cc:237] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.297681 843 catalog_manager.cc:5495] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 2 to 3, leader changed from ded90ff3c146407a822cc07a683e33b9 (127.0.203.1) to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 3 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:42.460018 1239 raft_consensus.cc:979] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01: : Instructing follower b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.461170 1276 raft_consensus.cc:1067] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 LEADER]: Signalling peer b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.462818 1022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:39186
I20260924 00:20:42.463282 1022 raft_consensus.cc:482] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:42.463557 1022 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:42.463861 1277 raft_consensus.cc:979] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01: : Instructing follower b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.464285 1276 raft_consensus.cc:1067] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 LEADER]: Signalling peer b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.465735 1020 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:39186
I20260924 00:20:42.466185 1020 raft_consensus.cc:482] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:42.466482 1020 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.467969 1022 raft_consensus.cc:504] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:42.469374 1022 leader_election.cc:288] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 4 election: Requested vote from peers 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697), ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173)
I20260924 00:20:42.470208 1094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:42.470393 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:42.470779 1094 raft_consensus.cc:3032] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:20:42.470975 946 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:42.471087 1094 raft_consensus.cc:729] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:42.471730 1020 raft_consensus.cc:504] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.471882 1094 consensus_queue.cc:260] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:42.473067 1094 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:42.473763 1020 leader_election.cc:288] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:42.474613 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:42.475116 947 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.476276 946 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 4.
I20260924 00:20:42.474978 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:42.477418 1095 raft_consensus.cc:3032] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:20:42.477567 988 leader_election.cc:302] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:42.477743 1095 raft_consensus.cc:729] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:42.478499 1263 raft_consensus.cc:2781] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:20:42.478971 1263 raft_consensus.cc:686] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:42.479393 1094 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 4.
I20260924 00:20:42.479252 1095 consensus_queue.cc:260] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.479835 1263 consensus_queue.cc:237] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:42.480576 1095 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.481594 947 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.483479 988 leader_election.cc:302] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:42.484575 1270 raft_consensus.cc:2781] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:20:42.486128 1270 raft_consensus.cc:686] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:42.487085 1095 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.486963 1270 consensus_queue.cc:237] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.488333 843 catalog_manager.cc:5495] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 3 to 4, leader changed from 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3) to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 4 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:42.500450 843 catalog_manager.cc:5495] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 2 to 3, leader changed from 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3) to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 3 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:42.547000 1269 raft_consensus.cc:979] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9: : Instructing follower b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.547492 1266 raft_consensus.cc:1067] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Signalling peer b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.549062 1020 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:39196
I20260924 00:20:42.549567 1020 raft_consensus.cc:482] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:42.549837 1020 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.554171 1020 raft_consensus.cc:504] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.555703 1020 leader_election.cc:288] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:42.556504 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:42.556784 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20b6a7dd2ed4f9da1c679eadf285605" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:42.557021 947 raft_consensus.cc:3032] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:20:42.557442 947 raft_consensus.cc:729] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:42.558506 1095 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.558132 947 consensus_queue.cc:260] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.559661 947 raft_consensus.cc:3037] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.563086 1095 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.563370 947 raft_consensus.cc:2454] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.564155 986 leader_election.cc:302] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:42.564970 1270 raft_consensus.cc:2781] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:20:42.565412 1270 raft_consensus.cc:686] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:42.566183 1270 consensus_queue.cc:237] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.572834 843 catalog_manager.cc:5495] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 2 to 3, leader changed from ded90ff3c146407a822cc07a683e33b9 (127.0.203.1) to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 3 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:42.578848 1269 raft_consensus.cc:979] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9: : Instructing follower b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.579391 1266 raft_consensus.cc:1067] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Signalling peer b01fcc3af0f740529463824ade56fcc7 to start an election
I20260924 00:20:42.581079 1020 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
from {username='slave'} at 127.0.0.1:39196
I20260924 00:20:42.581656 1020 raft_consensus.cc:482] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:42.581928 1020 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.586516 1020 raft_consensus.cc:504] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.588025 1020 leader_election.cc:288] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:42.589035 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:42.588999 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "b01fcc3af0f740529463824ade56fcc7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:42.589695 1095 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.589797 947 raft_consensus.cc:3032] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:20:42.590142 947 raft_consensus.cc:729] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:42.590672 947 consensus_queue.cc:260] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.591660 947 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:42.594343 1095 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.595278 986 leader_election.cc:302] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, b01fcc3af0f740529463824ade56fcc7; no voters:
I20260924 00:20:42.595453 947 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01fcc3af0f740529463824ade56fcc7 in term 3.
I20260924 00:20:42.596069 1270 raft_consensus.cc:2781] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:20:42.596544 1270 raft_consensus.cc:686] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Becoming Leader. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:42.597314 1270 consensus_queue.cc:237] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:42.603327 843 catalog_manager.cc:5495] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 reported cstate change: term changed from 2 to 3, leader changed from ded90ff3c146407a822cc07a683e33b9 (127.0.203.1) to b01fcc3af0f740529463824ade56fcc7 (127.0.203.2). New cstate: current_term: 3 leader_uuid: "b01fcc3af0f740529463824ade56fcc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:42.679214 1095 raft_consensus.cc:1261] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:42.681954 1270 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:20:42.815423 947 raft_consensus.cc:1261] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:42.937186 1263 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:20:42.956295 1094 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:42.957808 1270 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:20:42.967540 947 raft_consensus.cc:1261] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:42.968907 1286 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:20:42.980003 946 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:42.981647 1279 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:20:42.985574 946 raft_consensus.cc:1261] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:42.987609 1279 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:20:42.991806 1094 raft_consensus.cc:1261] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:42.994714 1270 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:20:43.006557 1095 raft_consensus.cc:1261] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:43.007988 1279 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:20:43.036182 946 raft_consensus.cc:1261] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:43.037738 1270 consensus_queue.cc:1035] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:20:43.048316 1095 raft_consensus.cc:1261] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Refusing update from remote peer b01fcc3af0f740529463824ade56fcc7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:43.049561 1263 consensus_queue.cc:1035] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260924 00:20:43.758517 968 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea060 after lost signal to thread 815
W20260924 00:20:43.759346 968 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea740 after lost signal to thread 878
W20260924 00:20:43.760061 968 debug-util.cc:398] Leaking SignalData structure 0x7b08001e8c60 after lost signal to thread 971
W20260924 00:20:43.760710 968 debug-util.cc:398] Leaking SignalData structure 0x7b0800195ba0 after lost signal to thread 1046
W20260924 00:20:43.761411 968 debug-util.cc:398] Leaking SignalData structure 0x7b0800198260 after lost signal to thread 1120
I20260924 00:20:45.272346 812 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260924 00:20:45.273125 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:45.280248 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26273297bebf41048141ce34242ae0d5"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:39220
I20260924 00:20:45.281395 1020 raft_consensus.cc:595] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 LEADER]: Received request to transfer leadership to TS ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:45.284415 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:39220
I20260924 00:20:45.285008 1020 raft_consensus.cc:595] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Received request to transfer leadership to TS ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:45.287660 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:39220
I20260924 00:20:45.288166 1020 raft_consensus.cc:595] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Received request to transfer leadership to TS ded90ff3c146407a822cc07a683e33b9
I20260924 00:20:45.290403 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:39220
I20260924 00:20:45.290861 1020 raft_consensus.cc:595] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Received request to transfer leadership to TS 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:45.292896 1020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
mode: GRACEFUL
new_leader_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:39220
I20260924 00:20:45.293396 1020 raft_consensus.cc:595] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 LEADER]: Received request to transfer leadership to TS 1b4bbe0422db401db0a37c388d720c01
I20260924 00:20:45.294392 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:20:45.294823 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:45.503579 1300 raft_consensus.cc:979] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:45.504293 1299 raft_consensus.cc:1067] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 LEADER]: Signalling peer 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:45.505986 1095 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:35944
I20260924 00:20:45.506527 1095 raft_consensus.cc:482] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:45.506937 1095 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:20:45.512812 1095 raft_consensus.cc:504] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:45.515477 1095 leader_election.cc:288] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 5 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173)
I20260924 00:20:45.516489 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:45.516812 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3883c040d834e2da77d51c1b30cc0c6" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:45.517374 1020 raft_consensus.cc:3032] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:20:45.517436 947 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:20:45.517846 1020 raft_consensus.cc:729] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:45.518682 1020 consensus_queue.cc:260] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:45.519881 1020 raft_consensus.cc:3037] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:20:45.524408 947 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 5.
I20260924 00:20:45.524761 1020 raft_consensus.cc:2454] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 5.
I20260924 00:20:45.525797 1062 leader_election.cc:302] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:45.526687 1276 raft_consensus.cc:2781] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:20:45.527928 1276 raft_consensus.cc:686] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 5 LEADER]: Becoming Leader. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:45.528911 1276 consensus_queue.cc:237] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } }
I20260924 00:20:45.539059 843 catalog_manager.cc:5495] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 reported cstate change: term changed from 4 to 5, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "1b4bbe0422db401db0a37c388d720c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:45.551141 1300 raft_consensus.cc:979] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:45.551620 1300 raft_consensus.cc:1067] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Signalling peer ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:45.553617 947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:48922
I20260924 00:20:45.554196 947 raft_consensus.cc:482] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:45.554561 947 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.559386 947 raft_consensus.cc:504] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.561303 947 leader_election.cc:288] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 4 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:45.562325 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:45.562455 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9976c883f10e45ae95ee12dda05212b1" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:45.562980 1020 raft_consensus.cc:3032] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:20:45.563050 1095 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.563407 1020 raft_consensus.cc:729] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:45.564209 1020 consensus_queue.cc:260] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.565411 1020 raft_consensus.cc:3037] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.570132 1095 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 4.
I20260924 00:20:45.571750 911 leader_election.cc:302] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:45.573875 1020 raft_consensus.cc:2454] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 4.
I20260924 00:20:45.573968 1307 raft_consensus.cc:2781] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:20:45.575423 1307 raft_consensus.cc:686] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 4 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:45.576432 1307 consensus_queue.cc:237] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.580878 1300 raft_consensus.cc:979] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:45.581445 1299 raft_consensus.cc:1067] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Signalling peer ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:45.582734 947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:48922
I20260924 00:20:45.583175 947 raft_consensus.cc:482] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:45.583443 947 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.589049 947 raft_consensus.cc:504] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.590570 843 catalog_manager.cc:5495] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 3 to 4, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 4 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:45.591394 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:45.591506 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec4554cc9314cec92eb5b7c40797696" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:45.592029 1095 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.591993 1020 raft_consensus.cc:3032] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:20:45.592453 1020 raft_consensus.cc:729] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:45.593303 1020 consensus_queue.cc:260] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.594568 1020 raft_consensus.cc:3037] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.596139 947 leader_election.cc:288] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 4 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:45.601214 1095 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 4.
I20260924 00:20:45.602445 911 leader_election.cc:302] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:45.603328 1307 raft_consensus.cc:2781] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:20:45.603813 1307 raft_consensus.cc:686] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 4 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:45.604018 1299 raft_consensus.cc:979] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:45.603885 1020 raft_consensus.cc:2454] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 4.
I20260924 00:20:45.604480 1299 raft_consensus.cc:1067] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 LEADER]: Signalling peer ded90ff3c146407a822cc07a683e33b9 to start an election
I20260924 00:20:45.606088 947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26273297bebf41048141ce34242ae0d5"
dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
from {username='slave'} at 127.0.0.1:48922
I20260924 00:20:45.605628 1307 consensus_queue.cc:237] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.606532 947 raft_consensus.cc:482] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:45.606855 947 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:45.612022 947 raft_consensus.cc:504] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.613706 947 leader_election.cc:288] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 3 election: Requested vote from peers b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547), 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3:41697)
I20260924 00:20:45.614676 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:45.615338 1020 raft_consensus.cc:3032] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:20:45.615669 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26273297bebf41048141ce34242ae0d5" candidate_uuid: "ded90ff3c146407a822cc07a683e33b9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
I20260924 00:20:45.616227 1095 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:45.614694 843 catalog_manager.cc:5495] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 3 to 4, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 4 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:45.615685 1020 raft_consensus.cc:729] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:45.618229 1020 consensus_queue.cc:260] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.619566 1020 raft_consensus.cc:3037] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:20:45.625886 1020 raft_consensus.cc:2454] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 3.
I20260924 00:20:45.626276 1095 raft_consensus.cc:2454] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded90ff3c146407a822cc07a683e33b9 in term 3.
I20260924 00:20:45.627076 913 leader_election.cc:302] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01fcc3af0f740529463824ade56fcc7, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:45.627972 1307 raft_consensus.cc:2781] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:20:45.628474 1307 raft_consensus.cc:686] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 3 LEADER]: Becoming Leader. State: Replica: ded90ff3c146407a822cc07a683e33b9, State: Running, Role: LEADER
I20260924 00:20:45.629415 1307 consensus_queue.cc:237] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.636906 843 catalog_manager.cc:5495] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 reported cstate change: term changed from 2 to 3, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to ded90ff3c146407a822cc07a683e33b9 (127.0.203.1). New cstate: current_term: 3 leader_uuid: "ded90ff3c146407a822cc07a683e33b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:45.853259 1299 raft_consensus.cc:979] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: : Instructing follower 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:45.853753 1300 raft_consensus.cc:1067] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Signalling peer 1b4bbe0422db401db0a37c388d720c01 to start an election
I20260924 00:20:45.855388 1095 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2"
dest_uuid: "1b4bbe0422db401db0a37c388d720c01"
from {username='slave'} at 127.0.0.1:35944
I20260924 00:20:45.856021 1095 raft_consensus.cc:482] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:45.856423 1095 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.864058 1095 raft_consensus.cc:504] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.866195 1095 leader_election.cc:288] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 4 election: Requested vote from peers ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173), b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547)
I20260924 00:20:45.867110 947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ded90ff3c146407a822cc07a683e33b9"
I20260924 00:20:45.867225 1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a9dc94f8b84b6ab7e76e7aea7f27e2" candidate_uuid: "1b4bbe0422db401db0a37c388d720c01" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b01fcc3af0f740529463824ade56fcc7"
I20260924 00:20:45.867678 947 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.867745 1020 raft_consensus.cc:3032] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:20:45.868187 1020 raft_consensus.cc:729] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b01fcc3af0f740529463824ade56fcc7, State: Running, Role: LEADER
I20260924 00:20:45.869004 1020 consensus_queue.cc:260] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.870214 1020 raft_consensus.cc:3037] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:20:45.874266 947 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 4.
I20260924 00:20:45.875748 1062 leader_election.cc:302] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4bbe0422db401db0a37c388d720c01, ded90ff3c146407a822cc07a683e33b9; no voters:
I20260924 00:20:45.876575 1276 raft_consensus.cc:2781] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:20:45.877058 1276 raft_consensus.cc:686] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 4 LEADER]: Becoming Leader. State: Replica: 1b4bbe0422db401db0a37c388d720c01, State: Running, Role: LEADER
I20260924 00:20:45.878079 1020 raft_consensus.cc:2454] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4bbe0422db401db0a37c388d720c01 in term 4.
I20260924 00:20:45.877965 1276 consensus_queue.cc:237] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } }
I20260924 00:20:45.886669 843 catalog_manager.cc:5495] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 reported cstate change: term changed from 3 to 4, leader changed from b01fcc3af0f740529463824ade56fcc7 (127.0.203.2) to 1b4bbe0422db401db0a37c388d720c01 (127.0.203.3). New cstate: current_term: 4 leader_uuid: "1b4bbe0422db401db0a37c388d720c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 } health_report { overall_health: HEALTHY } } }
I20260924 00:20:45.925905 947 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 5 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:20:45.927650 1301 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:20:45.951956 1020 raft_consensus.cc:1261] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 5 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:20:45.953431 1276 consensus_queue.cc:1035] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:20:46.046012 1020 raft_consensus.cc:1261] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:46.047274 1307 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:20:46.057674 1095 raft_consensus.cc:1261] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:46.059535 1307 consensus_queue.cc:1035] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:20:46.070761 1094 raft_consensus.cc:1261] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:46.072734 1322 consensus_queue.cc:1035] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:20:46.089035 1020 raft_consensus.cc:1261] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:20:46.095451 1320 consensus_queue.cc:1035] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:20:46.132902 1020 raft_consensus.cc:1261] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:46.134223 1325 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:20:46.143554 1094 raft_consensus.cc:1261] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Refusing update from remote peer ded90ff3c146407a822cc07a683e33b9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:46.145462 1325 consensus_queue.cc:1035] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4bbe0422db401db0a37c388d720c01" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:20:46.295845 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:46.301981 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:46.302425 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:46.425926 1020 raft_consensus.cc:1261] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:46.427066 1301 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fcc3af0f740529463824ade56fcc7" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:20:46.436360 947 raft_consensus.cc:1261] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Refusing update from remote peer 1b4bbe0422db401db0a37c388d720c01: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:20:46.437942 1301 consensus_queue.cc:1035] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded90ff3c146407a822cc07a683e33b9" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:20:47.303181 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:47.308467 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:47.308923 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:48.309747 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:48.314333 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:48.314762 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:49.315564 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:49.320906 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:49.321408 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:50.322256 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:50.327023 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:50.327402 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:51.328205 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:51.332880 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:51.333362 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:52.334137 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:52.338683 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:52.338997 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:53.339712 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:53.344445 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:53.344784 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:54.345466 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:20:54.350826 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:20:54.351266 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:20:55.377267 812 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260924 00:20:55.377919 812 tablet_server.cc:178] TabletServer@127.0.203.1:0 shutting down...
I20260924 00:20:55.404883 812 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260924 00:20:55.405627 812 tablet_replica.cc:330] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.406388 812 raft_consensus.cc:2229] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.406982 812 raft_consensus.cc:2258] T 8a7bb05c9df540c09ccff1dce38119ae P ded90ff3c146407a822cc07a683e33b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.409808 812 tablet_replica.cc:330] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.410480 812 raft_consensus.cc:2229] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 3 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.411247 812 raft_consensus.cc:2258] T 26273297bebf41048141ce34242ae0d5 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.412787 812 tablet_replica.cc:330] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.413335 812 raft_consensus.cc:2229] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.413676 812 raft_consensus.cc:2258] T c20b6a7dd2ed4f9da1c679eadf285605 P ded90ff3c146407a822cc07a683e33b9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.415140 812 tablet_replica.cc:330] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.415563 812 raft_consensus.cc:2229] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 4 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.416237 812 raft_consensus.cc:2258] T 9976c883f10e45ae95ee12dda05212b1 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.417802 812 tablet_replica.cc:330] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.418183 812 raft_consensus.cc:2229] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.418514 812 raft_consensus.cc:2258] T 18cac6776aec4050ac6fbd87e3c3701e P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.419903 812 tablet_replica.cc:330] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.420300 812 raft_consensus.cc:2229] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.420706 812 raft_consensus.cc:2258] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.422185 812 tablet_replica.cc:330] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.422591 812 raft_consensus.cc:2229] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.422897 812 raft_consensus.cc:2258] T aa24aa2041174acb84bbec10cdeb698c P ded90ff3c146407a822cc07a683e33b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.424449 812 tablet_replica.cc:330] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.424829 812 raft_consensus.cc:2229] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.425251 812 raft_consensus.cc:2258] T e3883c040d834e2da77d51c1b30cc0c6 P ded90ff3c146407a822cc07a683e33b9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.426667 812 tablet_replica.cc:330] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9: stopping tablet replica
I20260924 00:20:55.427016 812 raft_consensus.cc:2229] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 4 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.427678 812 raft_consensus.cc:2258] T 8ec4554cc9314cec92eb5b7c40797696 P ded90ff3c146407a822cc07a683e33b9 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260924 00:20:55.572515 988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.1:38173: connect: Connection refused (error 111)
W20260924 00:20:55.576224 988 consensus_peers.cc:487] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 -> Peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Couldn't send request to peer ded90ff3c146407a822cc07a683e33b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.1:38173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 00:20:55.576303 1062 consensus_peers.cc:487] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 -> Peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Couldn't send request to peer ded90ff3c146407a822cc07a683e33b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.1:38173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:20:55.600065 812 tablet_server.cc:194] TabletServer@127.0.203.1:0 shutdown complete.
I20260924 00:20:55.617697 812 tablet_server.cc:178] TabletServer@127.0.203.2:0 shutting down...
W20260924 00:20:55.621704 1062 consensus_peers.cc:487] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 -> Peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Couldn't send request to peer ded90ff3c146407a822cc07a683e33b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.1:38173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 00:20:55.627697 988 consensus_peers.cc:487] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 -> Peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Couldn't send request to peer ded90ff3c146407a822cc07a683e33b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.1:38173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:20:55.649513 812 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260924 00:20:55.650334 812 tablet_replica.cc:330] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.651081 812 raft_consensus.cc:2229] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.651621 812 raft_consensus.cc:2258] T 9976c883f10e45ae95ee12dda05212b1 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.653383 812 tablet_replica.cc:330] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.653879 812 raft_consensus.cc:2229] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.654735 812 raft_consensus.cc:2258] T 8a7bb05c9df540c09ccff1dce38119ae P b01fcc3af0f740529463824ade56fcc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.656772 812 tablet_replica.cc:330] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.657400 812 raft_consensus.cc:2229] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 3 LEADER]: Raft consensus shutting down.
W20260924 00:20:55.658123 1062 consensus_peers.cc:487] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 -> Peer b01fcc3af0f740529463824ade56fcc7 (127.0.203.2:46547): Couldn't send request to peer b01fcc3af0f740529463824ade56fcc7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:20:55.658294 812 raft_consensus.cc:2258] T c20b6a7dd2ed4f9da1c679eadf285605 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.660032 812 tablet_replica.cc:330] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.660501 812 raft_consensus.cc:2229] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.660904 812 raft_consensus.cc:2258] T aa24aa2041174acb84bbec10cdeb698c P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.662918 812 tablet_replica.cc:330] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.663375 812 raft_consensus.cc:2229] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 2 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.664070 812 raft_consensus.cc:2258] T 18cac6776aec4050ac6fbd87e3c3701e P b01fcc3af0f740529463824ade56fcc7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.665685 812 tablet_replica.cc:330] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.666134 812 raft_consensus.cc:2229] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.666523 812 raft_consensus.cc:2258] T e3883c040d834e2da77d51c1b30cc0c6 P b01fcc3af0f740529463824ade56fcc7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.668057 812 tablet_replica.cc:330] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.668511 812 raft_consensus.cc:2229] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.668906 812 raft_consensus.cc:2258] T 26273297bebf41048141ce34242ae0d5 P b01fcc3af0f740529463824ade56fcc7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.670365 812 tablet_replica.cc:330] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.670830 812 raft_consensus.cc:2229] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.671201 812 raft_consensus.cc:2258] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.672633 812 tablet_replica.cc:330] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7: stopping tablet replica
I20260924 00:20:55.673034 812 raft_consensus.cc:2229] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.673477 812 raft_consensus.cc:2258] T 8ec4554cc9314cec92eb5b7c40797696 P b01fcc3af0f740529463824ade56fcc7 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260924 00:20:55.691994 1062 consensus_peers.cc:487] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 -> Peer ded90ff3c146407a822cc07a683e33b9 (127.0.203.1:38173): Couldn't send request to peer ded90ff3c146407a822cc07a683e33b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.1:38173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:20:55.700434 812 tablet_server.cc:194] TabletServer@127.0.203.2:0 shutdown complete.
I20260924 00:20:55.716878 812 tablet_server.cc:178] TabletServer@127.0.203.3:0 shutting down...
I20260924 00:20:55.743634 812 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260924 00:20:55.744267 812 tablet_replica.cc:330] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.744885 812 raft_consensus.cc:2229] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.745450 812 raft_consensus.cc:2258] T 9976c883f10e45ae95ee12dda05212b1 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.747197 812 tablet_replica.cc:330] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.747686 812 raft_consensus.cc:2229] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 2 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.748492 812 raft_consensus.cc:2258] T aa24aa2041174acb84bbec10cdeb698c P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.750312 812 tablet_replica.cc:330] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.750814 812 raft_consensus.cc:2229] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 4 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.751559 812 raft_consensus.cc:2258] T b9a9dc94f8b84b6ab7e76e7aea7f27e2 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.753274 812 tablet_replica.cc:330] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.753762 812 raft_consensus.cc:2229] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.754194 812 raft_consensus.cc:2258] T 26273297bebf41048141ce34242ae0d5 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.755728 812 tablet_replica.cc:330] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.756214 812 raft_consensus.cc:2229] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 5 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.756964 812 raft_consensus.cc:2258] T e3883c040d834e2da77d51c1b30cc0c6 P 1b4bbe0422db401db0a37c388d720c01 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.758538 812 tablet_replica.cc:330] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.758976 812 raft_consensus.cc:2229] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.759367 812 raft_consensus.cc:2258] T 18cac6776aec4050ac6fbd87e3c3701e P 1b4bbe0422db401db0a37c388d720c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.761518 812 tablet_replica.cc:330] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.761950 812 raft_consensus.cc:2229] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.762352 812 raft_consensus.cc:2258] T 8a7bb05c9df540c09ccff1dce38119ae P 1b4bbe0422db401db0a37c388d720c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.763805 812 tablet_replica.cc:330] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.764245 812 raft_consensus.cc:2229] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.764650 812 raft_consensus.cc:2258] T 8ec4554cc9314cec92eb5b7c40797696 P 1b4bbe0422db401db0a37c388d720c01 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.766184 812 tablet_replica.cc:330] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01: stopping tablet replica
I20260924 00:20:55.766659 812 raft_consensus.cc:2229] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:55.767088 812 raft_consensus.cc:2258] T c20b6a7dd2ed4f9da1c679eadf285605 P 1b4bbe0422db401db0a37c388d720c01 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.794085 812 tablet_server.cc:194] TabletServer@127.0.203.3:0 shutdown complete.
I20260924 00:20:55.811371 812 master.cc:537] Master@127.0.203.62:36271 shutting down...
I20260924 00:20:55.947281 812 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:20:55.947957 812 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:55.948303 812 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1daf06ac7fd8419f841222befa8e5d3d: stopping tablet replica
I20260924 00:20:55.967161 812 master.cc:552] Master@127.0.203.62:36271 shutdown complete.
1b4bbe0422db401db0a37c388d720c01 1
b01fcc3af0f740529463824ade56fcc7 4
ded90ff3c146407a822cc07a683e33b9 4
1b4bbe0422db401db0a37c388d720c01 3
b01fcc3af0f740529463824ade56fcc7 3
ded90ff3c146407a822cc07a683e33b9 3
1b4bbe0422db401db0a37c388d720c01 1
b01fcc3af0f740529463824ade56fcc7 8
ded90ff3c146407a822cc07a683e33b9 0
1b4bbe0422db401db0a37c388d720c01 3
b01fcc3af0f740529463824ade56fcc7 3
ded90ff3c146407a822cc07a683e33b9 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31345 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260924 00:20:56.009707 812 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.203.62:42997
I20260924 00:20:56.010748 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:56.016551 1349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:56.016695 1350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:56.018585 1352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.019948 812 server_base.cc:1034] running on GCE node
I20260924 00:20:56.021332 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:56.021613 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:56.021802 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209256021785 us; error 0 us; skew 500 ppm
I20260924 00:20:56.022567 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:56.026669 812 webserver.cc:460] Webserver started at http://127.0.203.62:39667/ using document root <none> and password file <none>
I20260924 00:20:56.027290 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:56.027554 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:56.027870 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:56.029373 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/master-0-root/instance:
uuid: "2e5227c91c0e4163a5da679ac1764175"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.035406 812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:20:56.039916 1357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.040841 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:20:56.041186 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/master-0-root
uuid: "2e5227c91c0e4163a5da679ac1764175"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.041512 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:56.082538 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:56.083814 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:56.121189 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:42997
I20260924 00:20:56.121361 1408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:42997 every 8 connection(s)
I20260924 00:20:56.124970 1409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.134105 1409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175: Bootstrap starting.
I20260924 00:20:56.138460 1409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.142629 1409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175: No bootstrap required, opened a new log
I20260924 00:20:56.144652 1409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5227c91c0e4163a5da679ac1764175" member_type: VOTER }
I20260924 00:20:56.145082 1409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:56.145375 1409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5227c91c0e4163a5da679ac1764175, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.145931 1409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5227c91c0e4163a5da679ac1764175" member_type: VOTER }
I20260924 00:20:56.146366 1409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:20:56.146617 1409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:20:56.146893 1409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:56.150589 1409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5227c91c0e4163a5da679ac1764175" member_type: VOTER }
I20260924 00:20:56.151082 1409 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5227c91c0e4163a5da679ac1764175; no voters:
I20260924 00:20:56.152195 1409 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:20:56.152634 1412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:56.153915 1412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 1 LEADER]: Becoming Leader. State: Replica: 2e5227c91c0e4163a5da679ac1764175, State: Running, Role: LEADER
I20260924 00:20:56.154500 1412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5227c91c0e4163a5da679ac1764175" member_type: VOTER }
I20260924 00:20:56.155131 1409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:20:56.159317 1413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e5227c91c0e4163a5da679ac1764175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5227c91c0e4163a5da679ac1764175" member_type: VOTER } }
I20260924 00:20:56.159696 1414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5227c91c0e4163a5da679ac1764175. Latest consensus state: current_term: 1 leader_uuid: "2e5227c91c0e4163a5da679ac1764175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5227c91c0e4163a5da679ac1764175" member_type: VOTER } }
I20260924 00:20:56.160146 1413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:56.160332 1414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:56.162221 1420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:20:56.167919 1420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:20:56.172279 812 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260924 00:20:56.178196 1420 catalog_manager.cc:1347] Generated new cluster ID: 00dba80b91d24dbda48c5c0868335d46
I20260924 00:20:56.178488 1420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:20:56.201809 1420 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:20:56.203351 1420 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:20:56.217437 1420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175: Generated new TSK 0
I20260924 00:20:56.218166 1420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:20:56.239334 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:56.245790 1430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:56.247083 1431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.248737 812 server_base.cc:1034] running on GCE node
W20260924 00:20:56.249028 1433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.250030 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:56.250217 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:56.250332 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209256250324 us; error 0 us; skew 500 ppm
I20260924 00:20:56.250824 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:56.253217 812 webserver.cc:460] Webserver started at http://127.0.203.1:38267/ using document root <none> and password file <none>
I20260924 00:20:56.253664 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:56.253818 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:56.254006 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:56.254990 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-0-root/instance:
uuid: "1053ed314371460e9d47952fe809723c"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.259725 812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 00:20:56.262686 1438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.263402 812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 00:20:56.263703 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-0-root
uuid: "1053ed314371460e9d47952fe809723c"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.263949 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:56.297933 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:56.299135 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:56.300520 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:56.302812 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:56.303059 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.303371 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:56.303633 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.343727 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:40171
I20260924 00:20:56.343883 1500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:40171 every 8 connection(s)
I20260924 00:20:56.348276 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:56.355363 1505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:56.356395 1506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.361840 812 server_base.cc:1034] running on GCE node
W20260924 00:20:56.362154 1508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.363164 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:56.363488 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:56.363708 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209256363692 us; error 0 us; skew 500 ppm
I20260924 00:20:56.364367 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:56.364854 1501 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42997
I20260924 00:20:56.365346 1501 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:56.366456 1501 heartbeater.cc:511] Master 127.0.203.62:42997 requested a full tablet report, sending...
I20260924 00:20:56.367997 812 webserver.cc:460] Webserver started at http://127.0.203.2:44083/ using document root <none> and password file <none>
I20260924 00:20:56.368588 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:56.368798 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:56.369067 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:56.369104 1374 ts_manager.cc:194] Registered new tserver with Master: 1053ed314371460e9d47952fe809723c (127.0.203.1:40171)
I20260924 00:20:56.370417 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-1-root/instance:
uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.371203 1374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54232
I20260924 00:20:56.374933 812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 00:20:56.378464 1513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.379215 812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 00:20:56.379550 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-1-root
uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.379868 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:56.433944 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:56.435166 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:56.436659 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:56.438845 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:56.439069 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.439312 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:56.439476 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.478953 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:41589
I20260924 00:20:56.479094 1575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:41589 every 8 connection(s)
I20260924 00:20:56.483777 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:56.491142 1579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:56.492353 1580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:56.493742 1582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.495327 812 server_base.cc:1034] running on GCE node
I20260924 00:20:56.496470 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:20:56.496723 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:20:56.496918 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209256496906 us; error 0 us; skew 500 ppm
I20260924 00:20:56.497455 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:20:56.498162 1576 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42997
I20260924 00:20:56.498598 1576 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:56.499353 1576 heartbeater.cc:511] Master 127.0.203.62:42997 requested a full tablet report, sending...
I20260924 00:20:56.500298 812 webserver.cc:460] Webserver started at http://127.0.203.3:42521/ using document root <none> and password file <none>
I20260924 00:20:56.500901 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:56.501101 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:56.501324 1374 ts_manager.cc:194] Registered new tserver with Master: 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:56.501427 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:56.502651 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-2-root/instance:
uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.502944 1374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54246
I20260924 00:20:56.507104 812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:20:56.510551 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.511324 812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 00:20:56.511619 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-2-root
uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-r18p"
I20260924 00:20:56.511885 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790209224507772-812-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:56.532532 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:56.533812 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:56.535161 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:56.537431 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:56.537659 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.537889 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:56.538051 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.578264 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:43049
I20260924 00:20:56.578367 1649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:43049 every 8 connection(s)
I20260924 00:20:56.593379 1650 heartbeater.cc:347] Connected to a master server at 127.0.203.62:42997
I20260924 00:20:56.593765 1650 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:56.594435 1650 heartbeater.cc:511] Master 127.0.203.62:42997 requested a full tablet report, sending...
I20260924 00:20:56.596274 1374 ts_manager.cc:194] Registered new tserver with Master: 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:56.596988 812 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015818113s
I20260924 00:20:56.597918 1374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54254
I20260924 00:20:56.599505 812 test_util.cc:268] Using random seed: -1831850050
I20260924 00:20:56.624425 1374 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54258:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:20:56.627210 1374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:20:56.716028 1466 tablet_service.cc:1467] Processing CreateTablet for tablet 37c813bce1364f54b49aa76468d29cf2 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260924 00:20:56.716892 1464 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea996fbcd904d0a8d334053aad21f6e (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260924 00:20:56.717407 1462 tablet_service.cc:1467] Processing CreateTablet for tablet fdbc6827e58a491b85734200a08c259b (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260924 00:20:56.717890 1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c813bce1364f54b49aa76468d29cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.718667 1464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea996fbcd904d0a8d334053aad21f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.720419 1459 tablet_service.cc:1467] Processing CreateTablet for tablet a034f41b715c4999a425f140bf2596d7 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260924 00:20:56.720963 1465 tablet_service.cc:1467] Processing CreateTablet for tablet 088ec2985f1e4c2bb6b5b475b3411865 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260924 00:20:56.721858 1459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a034f41b715c4999a425f140bf2596d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.722587 1465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088ec2985f1e4c2bb6b5b475b3411865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.723743 1458 tablet_service.cc:1467] Processing CreateTablet for tablet c5f66b416bcc44aab36d0139b39a5b86 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260924 00:20:56.727982 1460 tablet_service.cc:1467] Processing CreateTablet for tablet 84cc19914dd541f9977d6b759f6ae1a8 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260924 00:20:56.730381 1461 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2cd9e84f3c46679226b515576d07f8 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260924 00:20:56.720419 1463 tablet_service.cc:1467] Processing CreateTablet for tablet 953b0d55f78b47a291a345db172604b7 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260924 00:20:56.724992 1458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f66b416bcc44aab36d0139b39a5b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.734122 1457 tablet_service.cc:1467] Processing CreateTablet for tablet e0953f1c7d15425daf6292577384b885 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260924 00:20:56.735227 1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbc6827e58a491b85734200a08c259b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.735849 1457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0953f1c7d15425daf6292577384b885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.740278 1541 tablet_service.cc:1467] Processing CreateTablet for tablet 37c813bce1364f54b49aa76468d29cf2 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260924 00:20:56.741725 1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c813bce1364f54b49aa76468d29cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.741957 1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84cc19914dd541f9977d6b759f6ae1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.745947 1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2cd9e84f3c46679226b515576d07f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.746156 1540 tablet_service.cc:1467] Processing CreateTablet for tablet 088ec2985f1e4c2bb6b5b475b3411865 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260924 00:20:56.747195 1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088ec2985f1e4c2bb6b5b475b3411865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.749465 1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953b0d55f78b47a291a345db172604b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.750619 1539 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea996fbcd904d0a8d334053aad21f6e (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260924 00:20:56.752070 1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea996fbcd904d0a8d334053aad21f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.754462 1536 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2cd9e84f3c46679226b515576d07f8 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260924 00:20:56.750619 1538 tablet_service.cc:1467] Processing CreateTablet for tablet 953b0d55f78b47a291a345db172604b7 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260924 00:20:56.755895 1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2cd9e84f3c46679226b515576d07f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.756728 1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953b0d55f78b47a291a345db172604b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.758327 1535 tablet_service.cc:1467] Processing CreateTablet for tablet 84cc19914dd541f9977d6b759f6ae1a8 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260924 00:20:56.759724 1535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84cc19914dd541f9977d6b759f6ae1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.752863 1537 tablet_service.cc:1467] Processing CreateTablet for tablet fdbc6827e58a491b85734200a08c259b (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260924 00:20:56.762382 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbc6827e58a491b85734200a08c259b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.763504 1615 tablet_service.cc:1467] Processing CreateTablet for tablet 37c813bce1364f54b49aa76468d29cf2 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260924 00:20:56.764878 1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c813bce1364f54b49aa76468d29cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.767777 1607 tablet_service.cc:1467] Processing CreateTablet for tablet c5f66b416bcc44aab36d0139b39a5b86 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260924 00:20:56.769170 1607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f66b416bcc44aab36d0139b39a5b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.769083 1608 tablet_service.cc:1467] Processing CreateTablet for tablet a034f41b715c4999a425f140bf2596d7 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260924 00:20:56.770401 1608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a034f41b715c4999a425f140bf2596d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.767781 1609 tablet_service.cc:1467] Processing CreateTablet for tablet 84cc19914dd541f9977d6b759f6ae1a8 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260924 00:20:56.774541 1609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84cc19914dd541f9977d6b759f6ae1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.774870 1614 tablet_service.cc:1467] Processing CreateTablet for tablet 088ec2985f1e4c2bb6b5b475b3411865 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260924 00:20:56.774981 1606 tablet_service.cc:1467] Processing CreateTablet for tablet e0953f1c7d15425daf6292577384b885 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260924 00:20:56.776141 1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088ec2985f1e4c2bb6b5b475b3411865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.776826 1606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0953f1c7d15425daf6292577384b885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.778172 1610 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2cd9e84f3c46679226b515576d07f8 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260924 00:20:56.779385 1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d2cd9e84f3c46679226b515576d07f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.782260 1612 tablet_service.cc:1467] Processing CreateTablet for tablet 953b0d55f78b47a291a345db172604b7 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260924 00:20:56.783552 1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953b0d55f78b47a291a345db172604b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.786207 1613 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea996fbcd904d0a8d334053aad21f6e (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260924 00:20:56.787516 1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea996fbcd904d0a8d334053aad21f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.790263 1611 tablet_service.cc:1467] Processing CreateTablet for tablet fdbc6827e58a491b85734200a08c259b (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260924 00:20:56.791656 1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbc6827e58a491b85734200a08c259b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.798799 1534 tablet_service.cc:1467] Processing CreateTablet for tablet a034f41b715c4999a425f140bf2596d7 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260924 00:20:56.799382 1533 tablet_service.cc:1467] Processing CreateTablet for tablet c5f66b416bcc44aab36d0139b39a5b86 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260924 00:20:56.800225 1534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a034f41b715c4999a425f140bf2596d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.801837 1532 tablet_service.cc:1467] Processing CreateTablet for tablet e0953f1c7d15425daf6292577384b885 (DEFAULT_TABLE table=test-workload [id=d34b012e142649689717b4e77e89c2c6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260924 00:20:56.803133 1532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0953f1c7d15425daf6292577384b885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.805469 1533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f66b416bcc44aab36d0139b39a5b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.854934 1665 tablet_bootstrap.cc:492] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:56.893323 1667 tablet_bootstrap.cc:492] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:56.900171 1666 tablet_bootstrap.cc:492] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:56.906520 1666 tablet_bootstrap.cc:654] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.912626 1665 tablet_bootstrap.cc:654] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.913719 1666 tablet_bootstrap.cc:492] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:56.914173 1666 ts_tablet_manager.cc:1397] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260924 00:20:56.917245 1667 tablet_bootstrap.cc:654] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.917415 1666 raft_consensus.cc:348] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.918246 1666 raft_consensus.cc:374] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:56.918543 1666 raft_consensus.cc:729] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.919616 1666 consensus_queue.cc:260] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.925595 1650 heartbeater.cc:503] Master 127.0.203.62:42997 was elected leader, sending a full tablet report...
I20260924 00:20:56.928792 1666 ts_tablet_manager.cc:1428] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20260924 00:20:56.930594 1666 tablet_bootstrap.cc:492] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:56.931128 1667 tablet_bootstrap.cc:492] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:56.931640 1667 ts_tablet_manager.cc:1397] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.008s
I20260924 00:20:56.931484 1665 tablet_bootstrap.cc:492] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:56.934455 1665 ts_tablet_manager.cc:1397] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.010s
I20260924 00:20:56.934747 1667 raft_consensus.cc:348] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.935640 1667 raft_consensus.cc:374] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:56.936002 1667 raft_consensus.cc:729] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.936961 1667 consensus_queue.cc:260] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.937168 1665 raft_consensus.cc:348] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:56.937810 1666 tablet_bootstrap.cc:654] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.937976 1665 raft_consensus.cc:374] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:56.938388 1665 raft_consensus.cc:729] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.939239 1665 consensus_queue.cc:260] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:56.942138 1576 heartbeater.cc:503] Master 127.0.203.62:42997 was elected leader, sending a full tablet report...
I20260924 00:20:56.942584 1667 ts_tablet_manager.cc:1428] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 00:20:56.943607 1667 tablet_bootstrap.cc:492] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:56.950305 1667 tablet_bootstrap.cc:654] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.958268 1501 heartbeater.cc:503] Master 127.0.203.62:42997 was elected leader, sending a full tablet report...
I20260924 00:20:56.958937 1665 ts_tablet_manager.cc:1428] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.024s user 0.000s sys 0.015s
I20260924 00:20:56.959913 1665 tablet_bootstrap.cc:492] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:56.960572 1666 tablet_bootstrap.cc:492] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:56.961020 1666 ts_tablet_manager.cc:1397] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.007s
I20260924 00:20:56.964779 1671 raft_consensus.cc:482] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:56.963780 1666 raft_consensus.cc:348] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.965265 1671 raft_consensus.cc:504] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.965967 1666 raft_consensus.cc:374] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:56.966259 1666 raft_consensus.cc:729] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.966935 1666 consensus_queue.cc:260] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.972077 1666 ts_tablet_manager.cc:1428] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260924 00:20:56.972723 1671 leader_election.cc:288] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:56.973011 1666 tablet_bootstrap.cc:492] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:56.982592 1666 tablet_bootstrap.cc:654] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.984896 1667 tablet_bootstrap.cc:492] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:56.985410 1667 ts_tablet_manager.cc:1397] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.004s
I20260924 00:20:56.987793 1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:56.988271 1667 raft_consensus.cc:348] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.989020 1666 tablet_bootstrap.cc:492] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:56.988991 1667 raft_consensus.cc:374] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 00:20:56.989348 1589 leader_election.cc:341] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:56.989902 1666 ts_tablet_manager.cc:1397] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20260924 00:20:56.989763 1667 raft_consensus.cc:729] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.990751 1667 consensus_queue.cc:260] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.992938 1667 ts_tablet_manager.cc:1428] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 00:20:56.992570 1666 raft_consensus.cc:348] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.993343 1666 raft_consensus.cc:374] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:56.993635 1666 raft_consensus.cc:729] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.993840 1667 tablet_bootstrap.cc:492] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:56.994268 1666 consensus_queue.cc:260] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:56.996692 1666 ts_tablet_manager.cc:1428] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260924 00:20:56.998298 1666 tablet_bootstrap.cc:492] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:57.002627 1665 tablet_bootstrap.cc:654] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.004769 1666 tablet_bootstrap.cc:654] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.007401 1667 tablet_bootstrap.cc:654] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.011518 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
W20260924 00:20:57.013875 1591 leader_election.cc:341] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.014372 1591 leader_election.cc:302] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c; no voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:57.015379 1676 raft_consensus.cc:2726] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.016036 1666 tablet_bootstrap.cc:492] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:57.016510 1666 ts_tablet_manager.cc:1397] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20260924 00:20:57.018906 1667 tablet_bootstrap.cc:492] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.019064 1665 tablet_bootstrap.cc:492] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.019408 1667 ts_tablet_manager.cc:1397] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.007s
I20260924 00:20:57.019570 1665 ts_tablet_manager.cc:1397] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.001s
I20260924 00:20:57.019277 1666 raft_consensus.cc:348] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.020097 1666 raft_consensus.cc:374] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.020380 1666 raft_consensus.cc:729] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.021175 1666 consensus_queue.cc:260] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.022351 1665 raft_consensus.cc:348] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.022605 1667 raft_consensus.cc:348] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.023074 1665 raft_consensus.cc:374] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.023414 1667 raft_consensus.cc:374] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.023579 1666 ts_tablet_manager.cc:1428] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260924 00:20:57.023608 1665 raft_consensus.cc:729] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.023752 1667 raft_consensus.cc:729] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.024510 1666 tablet_bootstrap.cc:492] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:57.024845 1667 consensus_queue.cc:260] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.024856 1665 consensus_queue.cc:260] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.028518 1665 ts_tablet_manager.cc:1428] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 00:20:57.029415 1665 tablet_bootstrap.cc:492] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.034183 1667 ts_tablet_manager.cc:1428] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260924 00:20:57.034412 1666 tablet_bootstrap.cc:654] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.035229 1665 tablet_bootstrap.cc:654] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.035356 1667 tablet_bootstrap.cc:492] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:57.044734 1667 tablet_bootstrap.cc:654] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.053088 1665 tablet_bootstrap.cc:492] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.053102 1666 tablet_bootstrap.cc:492] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:57.053776 1666 ts_tablet_manager.cc:1397] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.007s
I20260924 00:20:57.053776 1665 ts_tablet_manager.cc:1397] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.007s
I20260924 00:20:57.055308 1667 tablet_bootstrap.cc:492] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.055807 1667 ts_tablet_manager.cc:1397] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20260924 00:20:57.056509 1665 raft_consensus.cc:348] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.057139 1665 raft_consensus.cc:374] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.057476 1665 raft_consensus.cc:729] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.056964 1666 raft_consensus.cc:348] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.057858 1666 raft_consensus.cc:374] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.058197 1666 raft_consensus.cc:729] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.058120 1665 consensus_queue.cc:260] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.058593 1667 raft_consensus.cc:348] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.059087 1666 consensus_queue.cc:260] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.059707 1667 raft_consensus.cc:374] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.060070 1667 raft_consensus.cc:729] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.060865 1667 consensus_queue.cc:260] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.061551 1666 ts_tablet_manager.cc:1428] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260924 00:20:57.062548 1666 tablet_bootstrap.cc:492] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:57.064778 1665 ts_tablet_manager.cc:1428] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20260924 00:20:57.065397 1672 raft_consensus.cc:482] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.065755 1665 tablet_bootstrap.cc:492] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.065977 1672 raft_consensus.cc:504] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.069244 1666 tablet_bootstrap.cc:654] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.071967 1665 tablet_bootstrap.cc:654] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.079197 1672 leader_election.cc:288] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.092928 1665 tablet_bootstrap.cc:492] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.093407 1665 ts_tablet_manager.cc:1397] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.007s
I20260924 00:20:57.095248 1667 ts_tablet_manager.cc:1428] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20260924 00:20:57.095304 1665 raft_consensus.cc:348] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.096303 1665 raft_consensus.cc:374] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.096688 1665 raft_consensus.cc:729] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.096863 1667 tablet_bootstrap.cc:492] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:57.097662 1665 consensus_queue.cc:260] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.100090 1665 ts_tablet_manager.cc:1428] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 00:20:57.106480 1665 tablet_bootstrap.cc:492] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.109648 1666 tablet_bootstrap.cc:492] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:57.110178 1666 ts_tablet_manager.cc:1397] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.048s user 0.030s sys 0.016s
I20260924 00:20:57.113152 1666 raft_consensus.cc:348] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.113968 1666 raft_consensus.cc:374] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.114305 1666 raft_consensus.cc:729] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.115128 1666 consensus_queue.cc:260] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.120286 1666 ts_tablet_manager.cc:1428] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260924 00:20:57.120426 1665 tablet_bootstrap.cc:654] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.121465 1666 tablet_bootstrap.cc:492] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:57.124622 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
W20260924 00:20:57.127960 1516 leader_election.cc:341] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.128228 1666 tablet_bootstrap.cc:654] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.130445 1667 tablet_bootstrap.cc:654] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.138541 1665 tablet_bootstrap.cc:492] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.139019 1665 ts_tablet_manager.cc:1397] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.012s
I20260924 00:20:57.140376 1666 tablet_bootstrap.cc:492] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:57.140905 1666 ts_tablet_manager.cc:1397] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.011s
I20260924 00:20:57.141271 1667 tablet_bootstrap.cc:492] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.141749 1667 ts_tablet_manager.cc:1397] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.005s
I20260924 00:20:57.142181 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.141714 1665 raft_consensus.cc:348] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.142985 1665 raft_consensus.cc:374] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.143318 1665 raft_consensus.cc:729] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.143388 1672 raft_consensus.cc:482] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.143735 1666 raft_consensus.cc:348] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:57.143973 1672 raft_consensus.cc:504] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.144102 1665 consensus_queue.cc:260] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.144556 1666 raft_consensus.cc:374] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.145052 1666 raft_consensus.cc:729] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.145720 1667 raft_consensus.cc:348] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.146430 1665 ts_tablet_manager.cc:1428] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260924 00:20:57.145954 1666 consensus_queue.cc:260] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:57.146521 1667 raft_consensus.cc:374] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.147192 1667 raft_consensus.cc:729] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.147352 1665 tablet_bootstrap.cc:492] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.148003 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.148752 1625 raft_consensus.cc:2454] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 0.
I20260924 00:20:57.150012 1516 leader_election.cc:302] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
W20260924 00:20:57.150187 1517 leader_election.cc:341] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.150640 1517 leader_election.cc:302] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fd889ee03954c5aaa4e17f8d2870d62; no voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:20:57.152729 1666 ts_tablet_manager.cc:1428] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260924 00:20:57.153307 1676 raft_consensus.cc:482] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.147956 1667 consensus_queue.cc:260] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.153769 1676 raft_consensus.cc:504] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.157243 1676 leader_election.cc:288] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:57.159772 1665 tablet_bootstrap.cc:654] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.160887 1693 raft_consensus.cc:2781] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.162181 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.146163 1672 leader_election.cc:288] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.162554 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.153780 1666 tablet_bootstrap.cc:492] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:57.173609 1673 raft_consensus.cc:482] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.183552 1696 raft_consensus.cc:482] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.164793 1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.184072 1697 raft_consensus.cc:482] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.185674 1673 raft_consensus.cc:504] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.186452 1697 raft_consensus.cc:504] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.187462 1696 raft_consensus.cc:504] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.188488 1476 raft_consensus.cc:2454] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 0.
I20260924 00:20:57.191291 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.191598 1696 leader_election.cc:288] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
W20260924 00:20:57.193236 1517 leader_election.cc:341] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.194664 1673 leader_election.cc:288] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.184581 1693 raft_consensus.cc:482] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:20:57.168197 1422 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260924 00:20:57.185704 1591 leader_election.cc:341] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:20:57.195968 1589 leader_election.cc:341] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.196746 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.196751 1693 raft_consensus.cc:3037] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.197657 1625 raft_consensus.cc:2454] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 0.
I20260924 00:20:57.197996 1666 tablet_bootstrap.cc:654] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.198876 1589 leader_election.cc:302] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c; no voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:57.199163 1516 leader_election.cc:302] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters: 1053ed314371460e9d47952fe809723c
I20260924 00:20:57.164398 1692 raft_consensus.cc:2726] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.200171 1694 raft_consensus.cc:2781] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.200637 1694 raft_consensus.cc:482] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.201087 1694 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.204082 1665 tablet_bootstrap.cc:492] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.204823 1665 ts_tablet_manager.cc:1397] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.015s
I20260924 00:20:57.200652 1676 raft_consensus.cc:2726] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.208041 1665 raft_consensus.cc:348] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.209028 1665 raft_consensus.cc:374] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.209450 1665 raft_consensus.cc:729] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.210314 1665 consensus_queue.cc:260] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.202092 1667 ts_tablet_manager.cc:1428] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.060s user 0.006s sys 0.009s
I20260924 00:20:57.212988 1665 ts_tablet_manager.cc:1428] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:20:57.214154 1665 tablet_bootstrap.cc:492] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.217412 1697 leader_election.cc:288] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.218415 1667 tablet_bootstrap.cc:492] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:57.221087 1665 tablet_bootstrap.cc:654] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.225040 1667 tablet_bootstrap.cc:654] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.227972 1676 raft_consensus.cc:482] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.228430 1676 raft_consensus.cc:504] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.230095 1693 raft_consensus.cc:504] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.230603 1676 leader_election.cc:288] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:57.230965 1665 tablet_bootstrap.cc:492] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.231518 1665 ts_tablet_manager.cc:1397] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260924 00:20:57.232131 1693 leader_election.cc:288] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.212421 1694 raft_consensus.cc:504] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.232760 1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.235232 1551 raft_consensus.cc:2454] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 0.
I20260924 00:20:57.235792 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.236402 1589 leader_election.cc:302] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.237309 1676 raft_consensus.cc:2781] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.237483 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.234391 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.237727 1676 raft_consensus.cc:482] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.238837 1676 raft_consensus.cc:3037] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.239542 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.240139 1475 raft_consensus.cc:2454] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 0.
I20260924 00:20:57.239631 1665 raft_consensus.cc:348] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.240697 1665 raft_consensus.cc:374] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.241099 1665 raft_consensus.cc:729] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.241580 1673 raft_consensus.cc:482] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:20:57.242183 1440 leader_election.cc:341] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.242045 1673 raft_consensus.cc:504] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.241887 1665 consensus_queue.cc:260] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.242894 1697 raft_consensus.cc:482] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.233238 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:20:57.244817 1476 raft_consensus.cc:3037] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.245306 1625 raft_consensus.cc:3037] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.245461 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.246426 1549 raft_consensus.cc:2454] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.247156 1666 tablet_bootstrap.cc:492] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:57.247646 1666 ts_tablet_manager.cc:1397] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.010s
I20260924 00:20:57.248039 1440 leader_election.cc:302] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.236640 1550 raft_consensus.cc:2379] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1053ed314371460e9d47952fe809723c in current term 1: Already voted for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in this term.
I20260924 00:20:57.244240 1673 leader_election.cc:288] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.251828 1666 raft_consensus.cc:348] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.252612 1666 raft_consensus.cc:374] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.252925 1666 raft_consensus.cc:729] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.255369 1666 consensus_queue.cc:260] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.259363 1666 ts_tablet_manager.cc:1428] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260924 00:20:57.260727 1666 tablet_bootstrap.cc:492] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:57.261415 1673 raft_consensus.cc:2781] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.261794 1673 raft_consensus.cc:482] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.262099 1673 raft_consensus.cc:3037] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.268610 1666 tablet_bootstrap.cc:654] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.275542 1665 ts_tablet_manager.cc:1428] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.039s user 0.009s sys 0.026s
I20260924 00:20:57.276631 1665 tablet_bootstrap.cc:492] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.278201 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c"
W20260924 00:20:57.280221 1517 leader_election.cc:341] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.280328 1625 raft_consensus.cc:2454] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 1.
I20260924 00:20:57.243355 1697 raft_consensus.cc:504] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.281586 1516 leader_election.cc:302] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.282999 1665 tablet_bootstrap.cc:654] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.283775 1693 raft_consensus.cc:2781] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.284379 1693 raft_consensus.cc:686] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:57.288205 1694 leader_election.cc:288] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.287546 1693 consensus_queue.cc:237] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.290060 1676 raft_consensus.cc:504] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.291419 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.286216 1697 leader_election.cc:288] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.294160 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:20:57.295285 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:57.285789 1476 raft_consensus.cc:2454] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 1.
I20260924 00:20:57.300661 1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.300565 1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.301474 1622 raft_consensus.cc:2379] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1053ed314371460e9d47952fe809723c in current term 1: Already voted for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in this term.
I20260924 00:20:57.301609 1623 raft_consensus.cc:2379] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1053ed314371460e9d47952fe809723c in current term 1: Already voted for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in this term.
I20260924 00:20:57.293015 1676 leader_election.cc:288] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:57.302419 1621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.297096 1673 raft_consensus.cc:504] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.306016 1621 raft_consensus.cc:2454] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.306674 1475 raft_consensus.cc:2379] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 36bc8e53f14a42319c2e8fcd5004ec6c in current term 1: Already voted for candidate 1053ed314371460e9d47952fe809723c in this term.
I20260924 00:20:57.307824 1441 leader_election.cc:302] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:57.299347 1697 raft_consensus.cc:482] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:20:57.300144 1440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:41589, user_credentials={real_user=slave}} blocked reactor thread for 52225.6us
I20260924 00:20:57.304464 1550 raft_consensus.cc:3037] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.310861 1697 raft_consensus.cc:504] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:57.312716 1440 leader_election.cc:302] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1053ed314371460e9d47952fe809723c; no voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:57.299993 1625 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.298530 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.313488 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.315662 1667 tablet_bootstrap.cc:492] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.316107 1667 ts_tablet_manager.cc:1397] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.007s
I20260924 00:20:57.318876 1667 raft_consensus.cc:348] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.319657 1667 raft_consensus.cc:374] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.319970 1667 raft_consensus.cc:729] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
W20260924 00:20:57.321296 1441 leader_election.cc:341] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.321732 1441 leader_election.cc:302] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1053ed314371460e9d47952fe809723c; no voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:57.322320 1703 raft_consensus.cc:2781] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.323452 1703 raft_consensus.cc:482] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.323872 1703 raft_consensus.cc:3037] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.323057 1697 leader_election.cc:288] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:57.326436 1697 raft_consensus.cc:2726] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.330935 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.331676 1624 raft_consensus.cc:2454] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.332463 1550 raft_consensus.cc:2454] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 1.
I20260924 00:20:57.333024 1441 leader_election.cc:302] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:57.333609 1589 leader_election.cc:302] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters: 1053ed314371460e9d47952fe809723c
I20260924 00:20:57.333674 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.334430 1676 raft_consensus.cc:2781] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.320719 1667 consensus_queue.cc:260] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.333988 1374 catalog_manager.cc:5495] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.335075 1676 raft_consensus.cc:686] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 LEADER]: Becoming Leader. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:20:57.341634 1666 tablet_bootstrap.cc:492] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:57.342206 1666 ts_tablet_manager.cc:1397] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.082s user 0.016s sys 0.011s
I20260924 00:20:57.342392 1676 consensus_queue.cc:237] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.335096 1704 raft_consensus.cc:2726] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.338022 1625 raft_consensus.cc:2454] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 1.
I20260924 00:20:57.345806 1666 raft_consensus.cc:348] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.339834 1667 ts_tablet_manager.cc:1428] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20260924 00:20:57.337110 1697 raft_consensus.cc:482] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.346424 1704 raft_consensus.cc:2781] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.347261 1697 raft_consensus.cc:504] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.347818 1667 tablet_bootstrap.cc:492] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:57.350646 1697 leader_election.cc:288] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.348976 1704 raft_consensus.cc:482] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.346987 1516 leader_election.cc:302] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters: 1053ed314371460e9d47952fe809723c
I20260924 00:20:57.351608 1704 raft_consensus.cc:3037] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.346540 1666 raft_consensus.cc:374] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.327560 1673 leader_election.cc:288] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.352473 1693 raft_consensus.cc:2781] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.353461 1666 raft_consensus.cc:729] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.354343 1693 raft_consensus.cc:686] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:57.356244 1693 consensus_queue.cc:237] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.356103 1666 consensus_queue.cc:260] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.357575 1665 tablet_bootstrap.cc:492] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.358150 1665 ts_tablet_manager.cc:1397] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.082s user 0.024s sys 0.023s
I20260924 00:20:57.360844 1666 ts_tablet_manager.cc:1428] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260924 00:20:57.361797 1666 tablet_bootstrap.cc:492] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Bootstrap starting.
I20260924 00:20:57.361624 1665 raft_consensus.cc:348] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.362489 1665 raft_consensus.cc:374] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.362951 1665 raft_consensus.cc:729] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.352837 1703 raft_consensus.cc:504] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.366729 1665 consensus_queue.cc:260] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.372457 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.372978 1667 tablet_bootstrap.cc:654] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.373122 1625 raft_consensus.cc:2454] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.380748 1665 ts_tablet_manager.cc:1428] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20260924 00:20:57.382946 1666 tablet_bootstrap.cc:654] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.384021 1373 catalog_manager.cc:5495] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3). New cstate: current_term: 1 leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: HEALTHY } } }
I20260924 00:20:57.388602 1441 leader_election.cc:302] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:57.389567 1673 raft_consensus.cc:2781] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.390084 1673 raft_consensus.cc:482] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.390508 1673 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.392370 1703 leader_election.cc:288] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.393446 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.394057 1625 raft_consensus.cc:3037] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:20:57.398844 1440 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:41589, user_credentials={real_user=slave}} blocked reactor thread for 86259.6us
I20260924 00:20:57.400403 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:57.405512 1550 raft_consensus.cc:2379] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1053ed314371460e9d47952fe809723c in current term 1: Already voted for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in this term.
I20260924 00:20:57.407428 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.414279 1665 tablet_bootstrap.cc:492] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.416286 1704 raft_consensus.cc:504] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:57.417888 1673 raft_consensus.cc:504] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.419178 1625 raft_consensus.cc:2454] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 1.
I20260924 00:20:57.420735 1704 leader_election.cc:288] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Requested vote from peers 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:57.421312 1441 leader_election.cc:302] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:57.422307 1704 raft_consensus.cc:2781] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.422848 1704 raft_consensus.cc:686] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:57.423090 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.423754 1625 raft_consensus.cc:3037] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.423872 1704 consensus_queue.cc:237] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.426411 1665 tablet_bootstrap.cc:654] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.425840 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.434132 1709 raft_consensus.cc:482] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.434921 1709 raft_consensus.cc:504] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.439050 1709 leader_election.cc:288] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.439471 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.440452 1624 raft_consensus.cc:2454] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.442121 1441 leader_election.cc:302] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
W20260924 00:20:57.442826 1440 leader_election.cc:341] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.442867 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:57.443470 1709 raft_consensus.cc:2781] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.443858 1709 raft_consensus.cc:482] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.444033 1440 leader_election.cc:302] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1053ed314371460e9d47952fe809723c; no voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:57.444221 1709 raft_consensus.cc:3037] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.445287 1673 leader_election.cc:288] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.447491 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.448110 1624 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.454644 1666 tablet_bootstrap.cc:492] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c: No bootstrap required, opened a new log
I20260924 00:20:57.455281 1666 ts_tablet_manager.cc:1397] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent bootstrapping tablet: real 0.094s user 0.012s sys 0.007s
I20260924 00:20:57.450495 1673 raft_consensus.cc:2726] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.458729 1666 raft_consensus.cc:348] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.459726 1666 raft_consensus.cc:374] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.460084 1666 raft_consensus.cc:729] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.461073 1666 consensus_queue.cc:260] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.464628 1666 ts_tablet_manager.cc:1428] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c: Time spent starting tablet: real 0.009s user 0.008s sys 0.003s
I20260924 00:20:57.451974 1625 raft_consensus.cc:2454] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 1.
I20260924 00:20:57.468015 1441 leader_election.cc:302] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:57.468966 1673 raft_consensus.cc:2781] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election won for term 1
W20260924 00:20:57.469313 1440 leader_election.cc:341] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.469590 1673 raft_consensus.cc:686] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:57.470755 1673 consensus_queue.cc:237] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
W20260924 00:20:57.470678 1440 leader_election.cc:341] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:20:57.476686 1440 leader_election.cc:341] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.482527 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:57.483134 1667 tablet_bootstrap.cc:492] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.484200 1667 ts_tablet_manager.cc:1397] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.137s user 0.013s sys 0.035s
I20260924 00:20:57.484177 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:57.488103 1667 raft_consensus.cc:348] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.492173 1665 tablet_bootstrap.cc:492] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.492895 1665 ts_tablet_manager.cc:1397] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.079s user 0.022s sys 0.018s
I20260924 00:20:57.493811 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.489274 1667 raft_consensus.cc:374] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.494661 1667 raft_consensus.cc:729] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.495654 1693 raft_consensus.cc:482] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.495496 1667 consensus_queue.cc:260] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.496600 1550 raft_consensus.cc:2454] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.497685 1693 raft_consensus.cc:504] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.501190 1693 leader_election.cc:288] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.502652 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.502692 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
W20260924 00:20:57.498386 1440 leader_election.cc:341] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.504328 1625 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 0.
I20260924 00:20:57.504551 1475 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 0.
I20260924 00:20:57.506099 1665 raft_consensus.cc:348] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.506973 1665 raft_consensus.cc:374] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.507309 1665 raft_consensus.cc:729] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
W20260924 00:20:57.510113 1440 leader_election.cc:341] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:20:57.510161 1517 leader_election.cc:302] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.508108 1665 consensus_queue.cc:260] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.511674 1693 raft_consensus.cc:2781] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.512033 1693 raft_consensus.cc:482] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.512367 1693 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.513250 1665 ts_tablet_manager.cc:1428] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.020s user 0.004s sys 0.003s
I20260924 00:20:57.514317 1665 tablet_bootstrap.cc:492] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: Bootstrap starting.
I20260924 00:20:57.516847 1624 raft_consensus.cc:2454] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 1.
I20260924 00:20:57.517921 1441 leader_election.cc:302] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters: 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:57.518728 1673 raft_consensus.cc:2781] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.519857 1667 ts_tablet_manager.cc:1428] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.035s user 0.004s sys 0.004s
I20260924 00:20:57.521346 1673 raft_consensus.cc:686] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:57.522701 1673 consensus_queue.cc:237] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.522822 1374 catalog_manager.cc:5495] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.524752 1667 tablet_bootstrap.cc:492] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:57.526253 1665 tablet_bootstrap.cc:654] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.525908 1372 catalog_manager.cc:5495] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 1 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.531517 1693 raft_consensus.cc:504] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.536932 1667 tablet_bootstrap.cc:654] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.542842 1665 tablet_bootstrap.cc:492] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: No bootstrap required, opened a new log
I20260924 00:20:57.543365 1665 ts_tablet_manager.cc:1397] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.011s
I20260924 00:20:57.546382 1665 raft_consensus.cc:348] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.547348 1665 raft_consensus.cc:374] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.547714 1665 raft_consensus.cc:729] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.548660 1665 consensus_queue.cc:260] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.551882 1665 ts_tablet_manager.cc:1428] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260924 00:20:57.552325 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:20:57.552923 1475 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.555408 1667 tablet_bootstrap.cc:492] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.555955 1667 ts_tablet_manager.cc:1397] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.013s
I20260924 00:20:57.557796 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.558621 1624 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.558776 1667 raft_consensus.cc:348] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:57.559624 1667 raft_consensus.cc:374] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.559935 1667 raft_consensus.cc:729] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.560638 1667 consensus_queue.cc:260] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:20:57.561360 1475 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 1.
I20260924 00:20:57.562645 1517 leader_election.cc:302] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.563350 1667 ts_tablet_manager.cc:1428] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:20:57.563525 1694 raft_consensus.cc:2781] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.564370 1667 tablet_bootstrap.cc:492] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:57.565070 1694 raft_consensus.cc:686] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:57.565251 1693 leader_election.cc:288] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.567018 1694 consensus_queue.cc:237] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.570812 1667 tablet_bootstrap.cc:654] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.573453 1709 raft_consensus.cc:504] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.578238 1624 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 1.
I20260924 00:20:57.581197 1709 leader_election.cc:288] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.578225 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:57.583829 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.584141 1550 raft_consensus.cc:3037] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.584388 1624 raft_consensus.cc:3037] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.586827 1667 tablet_bootstrap.cc:492] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.587249 1667 ts_tablet_manager.cc:1397] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.008s
I20260924 00:20:57.589774 1667 raft_consensus.cc:348] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.590466 1667 raft_consensus.cc:374] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.590728 1667 raft_consensus.cc:729] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.591399 1667 consensus_queue.cc:260] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.592423 1624 raft_consensus.cc:2454] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 1.
I20260924 00:20:57.593756 1441 leader_election.cc:302] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:57.594147 1667 ts_tablet_manager.cc:1428] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260924 00:20:57.594708 1709 raft_consensus.cc:2781] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.595044 1667 tablet_bootstrap.cc:492] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62: Bootstrap starting.
I20260924 00:20:57.594964 1550 raft_consensus.cc:2454] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 1.
I20260924 00:20:57.595252 1709 raft_consensus.cc:686] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:57.596212 1709 consensus_queue.cc:237] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.598822 1372 catalog_manager.cc:5495] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 1 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.600234 1667 tablet_bootstrap.cc:654] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:57.601258 1372 catalog_manager.cc:5495] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 1 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.604653 1667 tablet_bootstrap.cc:492] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62: No bootstrap required, opened a new log
I20260924 00:20:57.605021 1667 ts_tablet_manager.cc:1397] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 00:20:57.606729 1667 raft_consensus.cc:348] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.607391 1667 raft_consensus.cc:374] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:57.607663 1667 raft_consensus.cc:729] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Initialized, Role: FOLLOWER
I20260924 00:20:57.608349 1667 consensus_queue.cc:260] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.610494 1667 ts_tablet_manager.cc:1428] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260924 00:20:57.611827 1374 catalog_manager.cc:5495] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.613018 1673 raft_consensus.cc:482] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.613597 1673 raft_consensus.cc:504] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.615480 1673 leader_election.cc:288] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.617219 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.617498 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.617902 1550 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.618227 1624 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 0.
I20260924 00:20:57.618975 1440 leader_election.cc:302] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.620023 1673 raft_consensus.cc:2781] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.620281 1673 raft_consensus.cc:482] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.620540 1673 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.623500 1372 catalog_manager.cc:5495] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 1 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.625519 1673 raft_consensus.cc:504] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.627285 1673 leader_election.cc:288] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.628038 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:57.628603 1550 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.629041 1694 raft_consensus.cc:482] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.629667 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.629559 1694 raft_consensus.cc:504] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.630218 1624 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.631484 1694 leader_election.cc:288] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.632920 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.634671 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.635190 1550 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 1.
I20260924 00:20:57.635452 1625 raft_consensus.cc:2379] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fd889ee03954c5aaa4e17f8d2870d62 in current term 1: Already voted for candidate 1053ed314371460e9d47952fe809723c in this term.
I20260924 00:20:57.635788 1624 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 1.
I20260924 00:20:57.636474 1440 leader_election.cc:302] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.636895 1517 leader_election.cc:302] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fd889ee03954c5aaa4e17f8d2870d62; no voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:20:57.637740 1694 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.638652 1673 raft_consensus.cc:2781] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.639163 1673 raft_consensus.cc:686] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:57.639967 1673 consensus_queue.cc:237] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.647368 1694 raft_consensus.cc:2726] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.649237 1372 catalog_manager.cc:5495] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 1 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.670732 1673 raft_consensus.cc:482] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.671177 1673 raft_consensus.cc:504] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.672933 1673 leader_election.cc:288] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.673902 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.673792 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" is_pre_election: true
I20260924 00:20:57.674588 1624 raft_consensus.cc:2379] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1053ed314371460e9d47952fe809723c in current term 1: Already voted for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in this term.
I20260924 00:20:57.675416 1441 leader_election.cc:302] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1053ed314371460e9d47952fe809723c; no voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:57.676044 1673 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.680377 1673 raft_consensus.cc:2726] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.743396 1694 raft_consensus.cc:482] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.743952 1694 raft_consensus.cc:504] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
W20260924 00:20:57.745404 1577 tablet.cc:2369] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:20:57.746032 1694 leader_election.cc:288] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.746510 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.746793 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.747515 1624 raft_consensus.cc:2379] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fd889ee03954c5aaa4e17f8d2870d62 in current term 1: Already voted for candidate 1053ed314371460e9d47952fe809723c in this term.
I20260924 00:20:57.748385 1516 leader_election.cc:302] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fd889ee03954c5aaa4e17f8d2870d62; no voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:20:57.748741 1694 raft_consensus.cc:482] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.749251 1694 raft_consensus.cc:504] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.749468 1693 raft_consensus.cc:3037] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.750983 1694 leader_election.cc:288] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.751863 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.751906 1676 consensus_queue.cc:1035] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:57.752619 1475 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 0.
I20260924 00:20:57.752556 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.753509 1624 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 0.
I20260924 00:20:57.753871 1517 leader_election.cc:302] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.754633 1694 raft_consensus.cc:2781] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:57.755041 1694 raft_consensus.cc:482] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:57.755362 1694 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.756661 1693 raft_consensus.cc:2726] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.763073 1694 raft_consensus.cc:504] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.765813 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:20:57.766402 1476 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.767246 1694 leader_election.cc:288] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.768813 1671 consensus_queue.cc:1035] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:57.766291 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:57.772300 1624 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.773159 1694 consensus_queue.cc:1035] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:57.781678 1476 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 1.
I20260924 00:20:57.782924 1624 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 1.
I20260924 00:20:57.783284 1517 leader_election.cc:302] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:57.784536 1694 raft_consensus.cc:2781] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:57.786988 1694 raft_consensus.cc:686] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:57.787859 1694 consensus_queue.cc:237] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.795225 1692 consensus_queue.cc:1035] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:57.798517 1372 catalog_manager.cc:5495] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 1 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:57.851579 1692 raft_consensus.cc:482] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:57.852109 1692 raft_consensus.cc:504] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:57.853826 1692 leader_election.cc:288] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:57.854923 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1053ed314371460e9d47952fe809723c" is_pre_election: true
I20260924 00:20:57.854945 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" is_pre_election: true
I20260924 00:20:57.856345 1624 raft_consensus.cc:2379] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7fd889ee03954c5aaa4e17f8d2870d62 in current term 1: Already voted for candidate 1053ed314371460e9d47952fe809723c in this term.
I20260924 00:20:57.857421 1516 leader_election.cc:302] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fd889ee03954c5aaa4e17f8d2870d62; no voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:20:57.858256 1692 raft_consensus.cc:3037] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:57.859417 1693 consensus_queue.cc:1035] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:57.864581 1692 raft_consensus.cc:2726] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:20:57.869338 1694 consensus_queue.cc:1035] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:57.875079 1673 consensus_queue.cc:1035] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:57.878830 1709 consensus_queue.cc:1035] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:57.887428 1709 consensus_queue.cc:1035] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:57.910410 1704 consensus_queue.cc:1035] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:57.936316 1625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:39214
I20260924 00:20:57.936923 1625 raft_consensus.cc:595] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:20:57.952580 1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:33570
I20260924 00:20:57.952517 1709 consensus_queue.cc:1035] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:57.953329 1550 raft_consensus.cc:595] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:20:57.965627 1692 consensus_queue.cc:1035] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:57.966288 1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:33570
I20260924 00:20:57.967000 1550 raft_consensus.cc:595] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:20:57.966928 1709 consensus_queue.cc:1035] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:57.978475 1694 consensus_queue.cc:1035] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:57.999188 1475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:35770
I20260924 00:20:57.999823 1475 raft_consensus.cc:595] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Received request to transfer leadership to TS 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:58.007501 1475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:35770
I20260924 00:20:58.008102 1475 raft_consensus.cc:595] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Received request to transfer leadership to TS 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:20:58.012586 1475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0953f1c7d15425daf6292577384b885"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:35770
I20260924 00:20:58.013108 1475 raft_consensus.cc:595] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Received request to transfer leadership to TS 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:20:58.047991 1704 consensus_queue.cc:1035] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:58.054373 1709 raft_consensus.cc:979] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c: : Instructing follower 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:20:58.054939 1709 raft_consensus.cc:1067] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Signalling peer 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:20:58.055100 1704 consensus_queue.cc:1035] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:58.061336 1550 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:56752
I20260924 00:20:58.062012 1550 raft_consensus.cc:482] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:58.062496 1550 raft_consensus.cc:3037] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.070104 1550 raft_consensus.cc:504] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.072585 1550 leader_election.cc:288] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 2 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:58.073345 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:58.073437 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f66b416bcc44aab36d0139b39a5b86" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:20:58.074065 1625 raft_consensus.cc:3037] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.074194 1475 raft_consensus.cc:3032] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:58.074636 1475 raft_consensus.cc:729] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:58.075403 1475 consensus_queue.cc:260] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.076189 1709 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:58.076642 1475 raft_consensus.cc:3037] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.081509 1625 raft_consensus.cc:2454] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 2.
I20260924 00:20:58.081214 1709 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:20:58.083943 1516 leader_election.cc:302] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:58.084784 1692 raft_consensus.cc:2781] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:58.085284 1692 raft_consensus.cc:686] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:58.086153 1692 consensus_queue.cc:237] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.087558 1475 raft_consensus.cc:2454] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 2.
I20260924 00:20:58.099643 1373 catalog_manager.cc:5495] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 1 to 2, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 2 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:58.204005 1732 raft_consensus.cc:979] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:20:58.204496 1732 raft_consensus.cc:1067] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:20:58.206202 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60490
I20260924 00:20:58.206811 1475 raft_consensus.cc:482] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:58.207243 1475 raft_consensus.cc:3037] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.212934 1475 raft_consensus.cc:504] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.214773 1475 leader_election.cc:288] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:58.215651 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:58.215729 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088ec2985f1e4c2bb6b5b475b3411865" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:58.216221 1550 raft_consensus.cc:3037] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.216290 1624 raft_consensus.cc:3032] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:58.216657 1624 raft_consensus.cc:729] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:20:58.217393 1624 consensus_queue.cc:260] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.218272 1624 raft_consensus.cc:3037] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.221472 1550 raft_consensus.cc:2454] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:20:58.222802 1440 leader_election.cc:302] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:58.223032 1624 raft_consensus.cc:2454] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:20:58.223538 1703 raft_consensus.cc:2781] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:58.224025 1703 raft_consensus.cc:686] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:58.224876 1703 consensus_queue.cc:237] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.231966 1374 catalog_manager.cc:5495] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 1 to 2, leader changed from 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 2 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:58.241466 1692 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:58.251425 1692 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:58.260622 1692 raft_consensus.cc:979] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:20:58.261173 1692 raft_consensus.cc:1067] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:20:58.262743 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60496
I20260924 00:20:58.263381 1475 raft_consensus.cc:482] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:58.263792 1475 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.269101 1475 raft_consensus.cc:504] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.270977 1475 leader_election.cc:288] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:58.271924 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:58.272149 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:58.272548 1550 raft_consensus.cc:3032] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:58.272704 1624 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.272915 1550 raft_consensus.cc:729] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:58.273752 1550 consensus_queue.cc:260] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.274673 1550 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.279183 1624 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:20:58.279748 1550 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:20:58.280274 1441 leader_election.cc:302] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:58.281244 1703 raft_consensus.cc:2781] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:58.281744 1703 raft_consensus.cc:686] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:58.282527 1703 consensus_queue.cc:237] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.288266 1371 catalog_manager.cc:5495] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 1 to 2, leader changed from 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 2 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:58.354717 1692 raft_consensus.cc:979] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:20:58.355165 1736 raft_consensus.cc:1067] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:20:58.356719 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60496
I20260924 00:20:58.357299 1475 raft_consensus.cc:482] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:58.357689 1475 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.362346 1475 raft_consensus.cc:504] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.365121 1475 leader_election.cc:288] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:58.367385 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:58.367908 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:58.368553 1625 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.368090 1550 raft_consensus.cc:3032] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:58.369300 1550 raft_consensus.cc:729] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:58.370172 1550 consensus_queue.cc:260] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.371448 1550 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.372639 1673 raft_consensus.cc:979] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c: : Instructing follower 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:20:58.373121 1703 raft_consensus.cc:1067] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Signalling peer 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:20:58.375186 1624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0953f1c7d15425daf6292577384b885"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:53206
I20260924 00:20:58.376159 1624 raft_consensus.cc:482] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:58.376598 1624 raft_consensus.cc:3037] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.377234 1625 raft_consensus.cc:2454] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:20:58.377815 1550 raft_consensus.cc:2454] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:20:58.378535 1441 leader_election.cc:302] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:20:58.379390 1703 raft_consensus.cc:2781] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:58.379817 1703 raft_consensus.cc:686] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:58.380576 1703 consensus_queue.cc:237] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.383279 1624 raft_consensus.cc:504] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.386013 1624 leader_election.cc:288] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 2 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:20:58.386490 1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:20:58.386893 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0953f1c7d15425daf6292577384b885" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:20:58.387184 1474 raft_consensus.cc:3032] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:58.387599 1550 raft_consensus.cc:3037] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.387600 1474 raft_consensus.cc:729] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:58.388749 1474 consensus_queue.cc:260] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.388963 1371 catalog_manager.cc:5495] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 1 to 2, leader changed from 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 2 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:58.390276 1474 raft_consensus.cc:3037] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.394215 1550 raft_consensus.cc:2454] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 2.
I20260924 00:20:58.395455 1589 leader_election.cc:302] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:58.396366 1732 raft_consensus.cc:2781] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:58.396839 1732 raft_consensus.cc:686] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Becoming Leader. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:20:58.397751 1732 consensus_queue.cc:237] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.399191 1474 raft_consensus.cc:2454] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 2.
I20260924 00:20:58.405481 1371 catalog_manager.cc:5495] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c reported cstate change: term changed from 1 to 2, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3). New cstate: current_term: 2 leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: HEALTHY } } }
I20260924 00:20:58.470633 1673 raft_consensus.cc:979] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c: : Instructing follower 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:20:58.471204 1703 raft_consensus.cc:1067] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Signalling peer 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:20:58.472752 1549 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:56752
I20260924 00:20:58.473275 1549 raft_consensus.cc:482] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:20:58.473582 1549 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.478466 1549 raft_consensus.cc:504] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.481320 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:20:58.480293 1549 leader_election.cc:288] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 2 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:20:58.481896 1624 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.481195 1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:20:58.482547 1474 raft_consensus.cc:3032] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:20:58.482970 1474 raft_consensus.cc:729] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:20:58.483917 1474 consensus_queue.cc:260] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.485105 1474 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:20:58.488857 1624 raft_consensus.cc:2454] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 2.
I20260924 00:20:58.490012 1516 leader_election.cc:302] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:20:58.490867 1694 raft_consensus.cc:2781] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:20:58.491362 1475 raft_consensus.cc:1261] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.491405 1694 raft_consensus.cc:686] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:20:58.491719 1474 raft_consensus.cc:2454] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 2.
I20260924 00:20:58.492486 1694 consensus_queue.cc:237] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:20:58.492851 1692 consensus_queue.cc:1035] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.500723 1624 raft_consensus.cc:1261] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.500881 1371 catalog_manager.cc:5495] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 1 to 2, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 2 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:58.503352 1693 consensus_queue.cc:1035] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.613330 1624 raft_consensus.cc:1261] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.614761 1703 consensus_queue.cc:1035] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.622536 1549 raft_consensus.cc:1261] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.623834 1709 consensus_queue.cc:1035] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.732278 1624 raft_consensus.cc:1261] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.733965 1709 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.741617 1549 raft_consensus.cc:1261] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.743427 1703 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.785326 1549 raft_consensus.cc:1261] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.786736 1732 consensus_queue.cc:1035] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.794870 1474 raft_consensus.cc:1261] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.796471 1732 consensus_queue.cc:1035] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.850948 1549 raft_consensus.cc:1261] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.852216 1703 consensus_queue.cc:1035] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.862469 1624 raft_consensus.cc:1261] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.863801 1709 consensus_queue.cc:1035] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.939685 1624 raft_consensus.cc:1261] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.940996 1693 consensus_queue.cc:1035] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:20:58.948799 1474 raft_consensus.cc:1261] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:20:58.950356 1693 consensus_queue.cc:1035] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:01.044808 812 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260924 00:21:01.045483 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:01.077499 1474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:35772
I20260924 00:21:01.078027 1474 raft_consensus.cc:595] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Received request to transfer leadership to TS 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:21:01.080006 1474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "953b0d55f78b47a291a345db172604b7"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:35772
I20260924 00:21:01.080500 1474 raft_consensus.cc:595] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Received request to transfer leadership to TS 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:21:01.081490 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260924 00:21:01.082051 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:01.534381 1709 raft_consensus.cc:979] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c: : Instructing follower 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:01.534866 1768 raft_consensus.cc:1067] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Signalling peer 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:01.536348 1624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:53206
I20260924 00:21:01.536824 1624 raft_consensus.cc:482] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:01.537081 1624 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:01.541209 1709 raft_consensus.cc:979] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: : Instructing follower 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:01.541656 1768 raft_consensus.cc:1067] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Signalling peer 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:01.542425 1624 raft_consensus.cc:504] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:01.543089 1625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "953b0d55f78b47a291a345db172604b7"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:53206
I20260924 00:21:01.543598 1625 raft_consensus.cc:482] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:01.543926 1625 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:01.544031 1624 leader_election.cc:288] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 3 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:21:01.545015 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:01.545015 1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:21:01.545738 1549 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:01.545771 1474 raft_consensus.cc:3032] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:01.546238 1474 raft_consensus.cc:729] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:01.546970 1474 consensus_queue.cc:260] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:01.548187 1474 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:01.548838 1625 raft_consensus.cc:504] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:01.550942 1625 leader_election.cc:288] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 2 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:21:01.551348 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:21:01.551589 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:01.551934 1475 raft_consensus.cc:3032] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:01.552187 1550 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:01.552304 1475 raft_consensus.cc:729] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:01.553021 1475 consensus_queue.cc:260] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:01.553887 1475 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:01.554663 1474 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 3.
I20260924 00:21:01.557111 1549 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 3.
I20260924 00:21:01.558370 1591 leader_election.cc:302] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:21:01.558730 1550 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 2.
I20260924 00:21:01.558952 1475 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 2.
I20260924 00:21:01.559190 1763 raft_consensus.cc:2781] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:01.560936 1589 leader_election.cc:302] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:21:01.561599 1763 raft_consensus.cc:686] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 LEADER]: Becoming Leader. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:21:01.562076 1769 raft_consensus.cc:2781] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:01.562513 1769 raft_consensus.cc:686] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Becoming Leader. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:21:01.563055 1769 consensus_queue.cc:237] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:01.562641 1763 consensus_queue.cc:237] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:01.573218 1371 catalog_manager.cc:5495] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c reported cstate change: term changed from 2 to 3, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3). New cstate: current_term: 3 leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:01.590921 1371 catalog_manager.cc:5495] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c reported cstate change: term changed from 1 to 2, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3). New cstate: current_term: 2 leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:01.981489 1550 raft_consensus.cc:1261] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:01.982677 1770 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:01.992098 1475 raft_consensus.cc:1261] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:01.994603 1769 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:02.015974 1550 raft_consensus.cc:1261] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:02.017257 1769 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:02.025282 1475 raft_consensus.cc:1261] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:02.026855 1763 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:02.082932 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:02.088421 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:02.088866 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:03.089681 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:03.095155 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:03.095572 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:04.096452 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:04.103785 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:04.104295 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:05.105455 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:05.111413 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:05.111837 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:06.112699 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:06.121052 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:06.121707 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:07.122568 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:07.127764 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:07.128216 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:08.129017 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:08.134204 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:08.134639 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:09.135417 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:09.140435 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:09.140872 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:10.141732 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:10.146735 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:10.147171 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:11.182929 812 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260924 00:21:11.192811 1625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:39214
I20260924 00:21:11.193554 1625 raft_consensus.cc:595] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:21:11.199070 1625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "953b0d55f78b47a291a345db172604b7"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:39214
I20260924 00:21:11.199781 1625 raft_consensus.cc:595] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:21:11.208338 1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:33570
I20260924 00:21:11.208787 1550 raft_consensus.cc:595] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:21:11.214114 1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:33570
I20260924 00:21:11.214593 1550 raft_consensus.cc:595] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:21:11.219784 1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
mode: GRACEFUL
new_leader_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:33570
I20260924 00:21:11.220269 1550 raft_consensus.cc:595] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Received request to transfer leadership to TS 1053ed314371460e9d47952fe809723c
I20260924 00:21:11.282137 1763 raft_consensus.cc:979] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.282693 1791 raft_consensus.cc:1067] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.284361 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60490
I20260924 00:21:11.284915 1475 raft_consensus.cc:482] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:11.285374 1475 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:11.291374 1475 raft_consensus.cc:504] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.293349 1475 leader_election.cc:288] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 4 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:21:11.294523 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:21:11.294312 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea996fbcd904d0a8d334053aad21f6e" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:11.295167 1550 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:11.295181 1625 raft_consensus.cc:3032] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:21:11.295662 1625 raft_consensus.cc:729] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:21:11.296335 1625 consensus_queue.cc:260] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.297433 1625 raft_consensus.cc:3037] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:11.300701 1550 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 4.
I20260924 00:21:11.302027 1440 leader_election.cc:302] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:21:11.302620 1625 raft_consensus.cc:2454] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 4.
I20260924 00:21:11.302889 1790 raft_consensus.cc:2781] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:21:11.304035 1790 raft_consensus.cc:686] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 4 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:11.305037 1790 consensus_queue.cc:237] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.312237 1373 catalog_manager.cc:5495] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 3 to 4, leader changed from 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 4 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:11.362473 1767 raft_consensus.cc:979] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.363010 1798 raft_consensus.cc:1067] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.364579 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60496
I20260924 00:21:11.365083 1475 raft_consensus.cc:482] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:11.365484 1475 raft_consensus.cc:3037] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:11.370137 1475 raft_consensus.cc:504] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.371547 1475 leader_election.cc:288] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:21:11.372547 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:11.372547 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a034f41b715c4999a425f140bf2596d7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:21:11.373440 1550 raft_consensus.cc:3032] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:11.373445 1625 raft_consensus.cc:3037] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:11.373936 1550 raft_consensus.cc:729] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:21:11.374759 1550 consensus_queue.cc:260] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.376117 1550 raft_consensus.cc:3037] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:11.380752 1625 raft_consensus.cc:2454] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:21:11.382119 1441 leader_election.cc:302] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:21:11.382778 1550 raft_consensus.cc:2454] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:21:11.383004 1790 raft_consensus.cc:2781] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:11.383543 1790 raft_consensus.cc:686] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:11.384648 1790 consensus_queue.cc:237] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.392400 1373 catalog_manager.cc:5495] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 1 to 2, leader changed from 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 2 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:11.460948 1767 raft_consensus.cc:979] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.461440 1798 raft_consensus.cc:1067] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.462808 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60496
I20260924 00:21:11.463272 1475 raft_consensus.cc:482] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:11.463609 1475 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:11.469228 1475 raft_consensus.cc:504] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.470793 1475 leader_election.cc:288] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 3 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:21:11.471346 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:11.471915 1550 raft_consensus.cc:3032] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:11.472205 1550 raft_consensus.cc:729] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:21:11.472317 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84cc19914dd541f9977d6b759f6ae1a8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:21:11.472865 1625 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:11.472954 1550 consensus_queue.cc:260] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.473912 1550 raft_consensus.cc:3037] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:11.477348 1625 raft_consensus.cc:2454] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 3.
I20260924 00:21:11.477897 1550 raft_consensus.cc:2454] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 3.
I20260924 00:21:11.478410 1441 leader_election.cc:302] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:21:11.479259 1790 raft_consensus.cc:2781] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:11.479697 1790 raft_consensus.cc:686] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 3 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:11.480604 1790 consensus_queue.cc:237] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.488343 1373 catalog_manager.cc:5495] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 2 to 3, leader changed from 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 3 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:11.604583 1763 raft_consensus.cc:979] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.604997 1791 raft_consensus.cc:1067] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.606410 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "953b0d55f78b47a291a345db172604b7"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60490
I20260924 00:21:11.606870 1475 raft_consensus.cc:482] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:11.607125 1475 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:11.611208 1475 raft_consensus.cc:504] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.612612 1475 leader_election.cc:288] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 3 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:21:11.613653 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:21:11.613687 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:11.614254 1625 raft_consensus.cc:3032] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:11.614293 1550 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:11.614519 1625 raft_consensus.cc:729] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:21:11.615170 1625 consensus_queue.cc:260] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.616017 1625 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:11.619766 1550 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 3.
I20260924 00:21:11.620716 1440 leader_election.cc:302] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:21:11.621309 1625 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 3.
I20260924 00:21:11.621810 1790 raft_consensus.cc:2781] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:11.622385 1790 raft_consensus.cc:686] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:11.623193 1790 consensus_queue.cc:237] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.630213 1373 catalog_manager.cc:5495] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 2 to 3, leader changed from 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 3 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:11.719972 1767 raft_consensus.cc:979] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62: : Instructing follower 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.720403 1798 raft_consensus.cc:1067] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Signalling peer 1053ed314371460e9d47952fe809723c to start an election
I20260924 00:21:11.722118 1475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8"
dest_uuid: "1053ed314371460e9d47952fe809723c"
from {username='slave'} at 127.0.0.1:60496
I20260924 00:21:11.722609 1475 raft_consensus.cc:482] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:11.722941 1475 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:11.727356 1475 raft_consensus.cc:504] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.728893 1475 leader_election.cc:288] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Requested vote from peers 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:21:11.729796 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:11.729913 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "1053ed314371460e9d47952fe809723c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:21:11.730481 1625 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:11.730471 1550 raft_consensus.cc:3032] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:11.731021 1550 raft_consensus.cc:729] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:21:11.731762 1550 consensus_queue.cc:260] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.732961 1550 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:11.737171 1625 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:21:11.738226 1441 leader_election.cc:302] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053ed314371460e9d47952fe809723c, 36bc8e53f14a42319c2e8fcd5004ec6c; no voters:
I20260924 00:21:11.738996 1790 raft_consensus.cc:2781] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:11.739065 1550 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053ed314371460e9d47952fe809723c in term 2.
I20260924 00:21:11.739563 1790 raft_consensus.cc:686] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Leader. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:11.740417 1790 consensus_queue.cc:237] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:11.743845 1550 raft_consensus.cc:1261] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:21:11.745338 1792 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:11.751549 1373 catalog_manager.cc:5495] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c reported cstate change: term changed from 1 to 2, leader changed from 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2) to 1053ed314371460e9d47952fe809723c (127.0.203.1). New cstate: current_term: 2 leader_uuid: "1053ed314371460e9d47952fe809723c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:11.759497 1625 raft_consensus.cc:1261] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:21:11.761492 1790 consensus_queue.cc:1035] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:11.886293 1550 raft_consensus.cc:1261] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:11.887468 1805 consensus_queue.cc:1035] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:11.898311 1625 raft_consensus.cc:1261] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:11.899685 1805 consensus_queue.cc:1035] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:11.923535 1625 raft_consensus.cc:1261] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:11.924664 1805 consensus_queue.cc:1035] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:11.933691 1550 raft_consensus.cc:1261] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:11.935133 1805 consensus_queue.cc:1035] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:12.079973 1550 raft_consensus.cc:1261] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:12.081179 1805 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:12.089403 1625 raft_consensus.cc:1261] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:12.091737 1805 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:12.189997 1550 raft_consensus.cc:1261] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:12.191218 1805 consensus_queue.cc:1035] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:12.199394 1625 raft_consensus.cc:1261] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Refusing update from remote peer 1053ed314371460e9d47952fe809723c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:12.201221 1792 consensus_queue.cc:1035] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:14.749894 812 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260924 00:21:14.750597 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:14.757987 1475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:35772
I20260924 00:21:14.758556 1475 raft_consensus.cc:595] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Received request to transfer leadership to TS 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:21:14.761226 1475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:35772
I20260924 00:21:14.761739 1475 raft_consensus.cc:595] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Received request to transfer leadership to TS 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:21:14.763990 1475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "953b0d55f78b47a291a345db172604b7"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:35772
I20260924 00:21:14.764516 1475 raft_consensus.cc:595] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 LEADER]: Received request to transfer leadership to TS 7fd889ee03954c5aaa4e17f8d2870d62
I20260924 00:21:14.766777 1475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b"
dest_uuid: "1053ed314371460e9d47952fe809723c"
mode: GRACEFUL
new_leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:35772
I20260924 00:21:14.767238 1475 raft_consensus.cc:595] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Received request to transfer leadership to TS 36bc8e53f14a42319c2e8fcd5004ec6c
I20260924 00:21:14.768183 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:21:14.768625 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:15.004887 1790 raft_consensus.cc:979] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: : Instructing follower 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:21:15.005714 1819 raft_consensus.cc:1067] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 LEADER]: Signalling peer 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:21:15.007822 1546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "953b0d55f78b47a291a345db172604b7"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:56752
I20260924 00:21:15.008677 1546 raft_consensus.cc:482] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:15.009279 1546 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:15.018415 1546 raft_consensus.cc:504] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.020912 1546 leader_election.cc:288] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 4 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:21:15.022256 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:21:15.022269 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953b0d55f78b47a291a345db172604b7" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:21:15.023041 1475 raft_consensus.cc:3032] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:21:15.023180 1625 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:15.023329 1475 raft_consensus.cc:729] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:15.024174 1475 consensus_queue.cc:260] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.025420 1475 raft_consensus.cc:3037] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:15.028882 1625 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 4.
I20260924 00:21:15.030174 1516 leader_election.cc:302] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:21:15.031250 1822 raft_consensus.cc:2781] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:21:15.031733 1822 raft_consensus.cc:686] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:21:15.032332 1475 raft_consensus.cc:2454] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 4.
I20260924 00:21:15.032847 1822 consensus_queue.cc:237] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.033669 1819 raft_consensus.cc:979] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c: : Instructing follower 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:15.034246 1790 raft_consensus.cc:1067] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Signalling peer 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:15.036015 1625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:53206
I20260924 00:21:15.036490 1625 raft_consensus.cc:482] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:15.036749 1625 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:15.044023 1625 raft_consensus.cc:504] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.046144 1625 leader_election.cc:288] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 3 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:21:15.047461 1546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:15.047492 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbc6827e58a491b85734200a08c259b" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:21:15.048949 1546 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:15.049029 1475 raft_consensus.cc:3032] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:15.049599 1475 raft_consensus.cc:729] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:15.050443 1475 consensus_queue.cc:260] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.051867 1475 raft_consensus.cc:3037] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:15.050908 1373 catalog_manager.cc:5495] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 3 to 4, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 4 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:15.055022 1546 raft_consensus.cc:2454] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 3.
I20260924 00:21:15.056463 1589 leader_election.cc:302] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:21:15.057253 1791 raft_consensus.cc:2781] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:15.057745 1791 raft_consensus.cc:686] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 LEADER]: Becoming Leader. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:21:15.058797 1791 consensus_queue.cc:237] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.059816 1475 raft_consensus.cc:2454] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 3.
I20260924 00:21:15.075870 1371 catalog_manager.cc:5495] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c reported cstate change: term changed from 2 to 3, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3). New cstate: current_term: 3 leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:15.170399 1790 raft_consensus.cc:979] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c: : Instructing follower 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:15.170856 1819 raft_consensus.cc:1067] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Signalling peer 36bc8e53f14a42319c2e8fcd5004ec6c to start an election
I20260924 00:21:15.172165 1625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2"
dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
from {username='slave'} at 127.0.0.1:53206
I20260924 00:21:15.172633 1625 raft_consensus.cc:482] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:15.172864 1625 raft_consensus.cc:3037] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:15.177466 1625 raft_consensus.cc:504] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:21:15.178917 1625 leader_election.cc:288] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 2 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589)
I20260924 00:21:15.179679 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:21:15.179905 1546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c813bce1364f54b49aa76468d29cf2" candidate_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
I20260924 00:21:15.180222 1475 raft_consensus.cc:3032] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:15.180531 1475 raft_consensus.cc:729] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:15.180421 1546 raft_consensus.cc:3037] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:15.181313 1475 consensus_queue.cc:260] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:21:15.182348 1475 raft_consensus.cc:3037] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:15.185495 1546 raft_consensus.cc:2454] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 2.
I20260924 00:21:15.186478 1589 leader_election.cc:302] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:21:15.187216 1791 raft_consensus.cc:2781] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:15.187311 1475 raft_consensus.cc:2454] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36bc8e53f14a42319c2e8fcd5004ec6c in term 2.
I20260924 00:21:15.187947 1791 raft_consensus.cc:686] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Becoming Leader. State: Replica: 36bc8e53f14a42319c2e8fcd5004ec6c, State: Running, Role: LEADER
I20260924 00:21:15.188977 1791 consensus_queue.cc:237] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } }
I20260924 00:21:15.195973 1371 catalog_manager.cc:5495] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c reported cstate change: term changed from 1 to 2, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3). New cstate: current_term: 2 leader_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:15.262478 1790 raft_consensus.cc:979] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c: : Instructing follower 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:21:15.263017 1819 raft_consensus.cc:1067] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Signalling peer 7fd889ee03954c5aaa4e17f8d2870d62 to start an election
I20260924 00:21:15.264509 1546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8"
dest_uuid: "7fd889ee03954c5aaa4e17f8d2870d62"
from {username='slave'} at 127.0.0.1:56752
I20260924 00:21:15.265035 1546 raft_consensus.cc:482] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:15.265419 1546 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:15.270193 1546 raft_consensus.cc:504] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.271838 1546 leader_election.cc:288] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 3 election: Requested vote from peers 1053ed314371460e9d47952fe809723c (127.0.203.1:40171), 36bc8e53f14a42319c2e8fcd5004ec6c (127.0.203.3:43049)
I20260924 00:21:15.272622 1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1053ed314371460e9d47952fe809723c"
I20260924 00:21:15.272940 1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2cd9e84f3c46679226b515576d07f8" candidate_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c"
I20260924 00:21:15.273284 1475 raft_consensus.cc:3032] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:15.273444 1625 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:15.273599 1475 raft_consensus.cc:729] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1053ed314371460e9d47952fe809723c, State: Running, Role: LEADER
I20260924 00:21:15.274365 1475 consensus_queue.cc:260] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.275547 1475 raft_consensus.cc:3037] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:15.279963 1625 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 3.
I20260924 00:21:15.281039 1516 leader_election.cc:302] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36bc8e53f14a42319c2e8fcd5004ec6c, 7fd889ee03954c5aaa4e17f8d2870d62; no voters:
I20260924 00:21:15.281946 1822 raft_consensus.cc:2781] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:15.281989 1475 raft_consensus.cc:2454] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd889ee03954c5aaa4e17f8d2870d62 in term 3.
I20260924 00:21:15.282472 1822 raft_consensus.cc:686] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 LEADER]: Becoming Leader. State: Replica: 7fd889ee03954c5aaa4e17f8d2870d62, State: Running, Role: LEADER
I20260924 00:21:15.283373 1822 consensus_queue.cc:237] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } }
I20260924 00:21:15.289901 1371 catalog_manager.cc:5495] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 reported cstate change: term changed from 2 to 3, leader changed from 1053ed314371460e9d47952fe809723c (127.0.203.1) to 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2). New cstate: current_term: 3 leader_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:15.451412 1546 raft_consensus.cc:1261] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:15.452852 1791 consensus_queue.cc:1035] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:15.463514 1475 raft_consensus.cc:1261] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:15.465519 1821 consensus_queue.cc:1035] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260924 00:21:15.558313 1497 debug-util.cc:398] Leaking SignalData structure 0x7b08001fb980 after lost signal to thread 815
W20260924 00:21:15.559101 1497 debug-util.cc:398] Leaking SignalData structure 0x7b08001fb6c0 after lost signal to thread 1408
W20260924 00:21:15.560014 1497 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2080 after lost signal to thread 1500
W20260924 00:21:15.560810 1497 debug-util.cc:398] Leaking SignalData structure 0x7b0800120940 after lost signal to thread 1575
W20260924 00:21:15.561694 1497 debug-util.cc:398] Leaking SignalData structure 0x7b08000876c0 after lost signal to thread 1649
I20260924 00:21:15.596514 1625 raft_consensus.cc:1261] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:21:15.597741 1822 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:15.607681 1475 raft_consensus.cc:1261] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 4 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:21:15.610350 1798 consensus_queue.cc:1035] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:15.736739 1475 raft_consensus.cc:1261] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:15.738037 1821 consensus_queue.cc:1035] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:15.748850 1546 raft_consensus.cc:1261] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Refusing update from remote peer 36bc8e53f14a42319c2e8fcd5004ec6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:15.750452 1791 consensus_queue.cc:1035] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd889ee03954c5aaa4e17f8d2870d62" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:15.756191 1625 raft_consensus.cc:1261] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:15.757661 1840 consensus_queue.cc:1035] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36bc8e53f14a42319c2e8fcd5004ec6c" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:15.769843 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:15.779203 1475 raft_consensus.cc:1261] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Refusing update from remote peer 7fd889ee03954c5aaa4e17f8d2870d62: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:21:15.782480 1845 consensus_queue.cc:1035] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1053ed314371460e9d47952fe809723c" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:15.783133 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:15.783833 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:16.784909 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:16.798022 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:16.798656 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:17.799799 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:17.808951 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:17.809633 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:18.930575 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:18.937129 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:18.937592 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260924 00:21:19.243953 1646 debug-util.cc:398] Leaking SignalData structure 0x7b08001f92c0 after lost signal to thread 815
W20260924 00:21:19.244722 1646 debug-util.cc:398] Leaking SignalData structure 0x7b08001b4c80 after lost signal to thread 1408
W20260924 00:21:19.245602 1646 debug-util.cc:398] Leaking SignalData structure 0x7b080008f4e0 after lost signal to thread 1500
W20260924 00:21:19.246404 1646 debug-util.cc:398] Leaking SignalData structure 0x7b08000dfae0 after lost signal to thread 1575
W20260924 00:21:19.247202 1646 debug-util.cc:398] Leaking SignalData structure 0x7b08000df780 after lost signal to thread 1649
I20260924 00:21:19.938449 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:19.944530 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:19.945032 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:20.945914 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:20.951967 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:20.952512 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:21.953392 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:21.959213 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:21.959731 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:22.960623 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:22.965955 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:22.966642 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:23.967492 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:23.973663 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260924 00:21:23.974151 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:25.012017 812 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260924 00:21:25.013058 812 tablet_server.cc:178] TabletServer@127.0.203.1:0 shutting down...
W20260924 00:21:25.015883 1517 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20260924 00:21:25.021272 1517 consensus_peers.cc:487] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 -> Peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Couldn't send request to peer 1053ed314371460e9d47952fe809723c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260924 00:21:25.022073 1591 consensus_peers.cc:487] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c -> Peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Couldn't send request to peer 1053ed314371460e9d47952fe809723c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260924 00:21:25.045857 1591 consensus_peers.cc:487] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c -> Peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Couldn't send request to peer 1053ed314371460e9d47952fe809723c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:25.061542 812 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260924 00:21:25.063304 812 tablet_replica.cc:330] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.065193 812 raft_consensus.cc:2229] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.066835 812 raft_consensus.cc:2258] T 953b0d55f78b47a291a345db172604b7 P 1053ed314371460e9d47952fe809723c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.070192 812 tablet_replica.cc:330] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.070986 812 raft_consensus.cc:2229] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.071642 812 raft_consensus.cc:2258] T c5f66b416bcc44aab36d0139b39a5b86 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.073827 812 tablet_replica.cc:330] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.074499 812 raft_consensus.cc:2229] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 3 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.075870 812 raft_consensus.cc:2258] T 84cc19914dd541f9977d6b759f6ae1a8 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.078009 812 tablet_replica.cc:330] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.078691 812 raft_consensus.cc:2229] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.079241 812 raft_consensus.cc:2258] T fdbc6827e58a491b85734200a08c259b P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.081274 812 tablet_replica.cc:330] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.081804 812 raft_consensus.cc:2229] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.082324 812 raft_consensus.cc:2258] T 4d2cd9e84f3c46679226b515576d07f8 P 1053ed314371460e9d47952fe809723c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.084177 812 tablet_replica.cc:330] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.084698 812 raft_consensus.cc:2229] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.085884 812 raft_consensus.cc:2258] T a034f41b715c4999a425f140bf2596d7 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.087715 812 tablet_replica.cc:330] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.088218 812 raft_consensus.cc:2229] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.088721 812 raft_consensus.cc:2258] T e0953f1c7d15425daf6292577384b885 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.090899 812 tablet_replica.cc:330] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.091403 812 raft_consensus.cc:2229] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 2 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.092509 812 raft_consensus.cc:2258] T 088ec2985f1e4c2bb6b5b475b3411865 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.094377 812 tablet_replica.cc:330] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.094877 812 raft_consensus.cc:2229] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 4 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.095927 812 raft_consensus.cc:2258] T 8ea996fbcd904d0a8d334053aad21f6e P 1053ed314371460e9d47952fe809723c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.098016 812 tablet_replica.cc:330] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c: stopping tablet replica
I20260924 00:21:25.098480 812 raft_consensus.cc:2229] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.098950 812 raft_consensus.cc:2258] T 37c813bce1364f54b49aa76468d29cf2 P 1053ed314371460e9d47952fe809723c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.138935 812 tablet_server.cc:194] TabletServer@127.0.203.1:0 shutdown complete.
I20260924 00:21:25.161675 812 tablet_server.cc:178] TabletServer@127.0.203.2:0 shutting down...
I20260924 00:21:25.192883 812 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260924 00:21:25.193754 812 tablet_replica.cc:330] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.194581 812 raft_consensus.cc:2229] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.195226 812 raft_consensus.cc:2258] T 84cc19914dd541f9977d6b759f6ae1a8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.197503 812 tablet_replica.cc:330] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.198107 812 raft_consensus.cc:2229] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.199206 812 raft_consensus.cc:2258] T c5f66b416bcc44aab36d0139b39a5b86 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.201273 812 tablet_replica.cc:330] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.201830 812 raft_consensus.cc:2229] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.202337 812 raft_consensus.cc:2258] T 37c813bce1364f54b49aa76468d29cf2 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.204088 812 tablet_replica.cc:330] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.204612 812 raft_consensus.cc:2229] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.205314 812 raft_consensus.cc:2258] T 8ea996fbcd904d0a8d334053aad21f6e P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.207083 812 tablet_replica.cc:330] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.207587 812 raft_consensus.cc:2229] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.208020 812 raft_consensus.cc:2258] T a034f41b715c4999a425f140bf2596d7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.209987 812 tablet_replica.cc:330] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.210515 812 raft_consensus.cc:2229] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.211072 812 raft_consensus.cc:2258] T fdbc6827e58a491b85734200a08c259b P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.213276 812 tablet_replica.cc:330] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.214166 812 raft_consensus.cc:2229] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.215161 812 raft_consensus.cc:2258] T 4d2cd9e84f3c46679226b515576d07f8 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.217342 812 tablet_replica.cc:330] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.217939 812 raft_consensus.cc:2229] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.219046 812 raft_consensus.cc:2258] T 953b0d55f78b47a291a345db172604b7 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.221181 812 tablet_replica.cc:330] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.221755 812 raft_consensus.cc:2229] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.222338 812 raft_consensus.cc:2258] T e0953f1c7d15425daf6292577384b885 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260924 00:21:25.224443 1591 consensus_peers.cc:487] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c -> Peer 1053ed314371460e9d47952fe809723c (127.0.203.1:40171): Couldn't send request to peer 1053ed314371460e9d47952fe809723c. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.1:40171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:25.224723 812 tablet_replica.cc:330] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62: stopping tablet replica
I20260924 00:21:25.225399 812 raft_consensus.cc:2229] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.225953 812 raft_consensus.cc:2258] T 088ec2985f1e4c2bb6b5b475b3411865 P 7fd889ee03954c5aaa4e17f8d2870d62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.257069 812 tablet_server.cc:194] TabletServer@127.0.203.2:0 shutdown complete.
I20260924 00:21:25.281225 812 tablet_server.cc:178] TabletServer@127.0.203.3:0 shutting down...
I20260924 00:21:25.314654 812 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260924 00:21:25.315265 812 tablet_replica.cc:330] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.316061 812 raft_consensus.cc:2229] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.316612 812 raft_consensus.cc:2258] T 953b0d55f78b47a291a345db172604b7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.318523 812 tablet_replica.cc:330] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.319083 812 raft_consensus.cc:2229] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.319526 812 raft_consensus.cc:2258] T 8ea996fbcd904d0a8d334053aad21f6e P 36bc8e53f14a42319c2e8fcd5004ec6c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.321354 812 tablet_replica.cc:330] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.321869 812 raft_consensus.cc:2229] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.322669 812 raft_consensus.cc:2258] T 37c813bce1364f54b49aa76468d29cf2 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.324344 812 tablet_replica.cc:330] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.324882 812 raft_consensus.cc:2229] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.325397 812 raft_consensus.cc:2258] T c5f66b416bcc44aab36d0139b39a5b86 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.327155 812 tablet_replica.cc:330] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.327757 812 raft_consensus.cc:2229] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.328356 812 raft_consensus.cc:2258] T 84cc19914dd541f9977d6b759f6ae1a8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.330340 812 tablet_replica.cc:330] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.330861 812 raft_consensus.cc:2229] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Raft consensus shutting down.
W20260924 00:21:25.331082 1589 consensus_peers.cc:487] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c -> Peer 7fd889ee03954c5aaa4e17f8d2870d62 (127.0.203.2:41589): Couldn't send request to peer 7fd889ee03954c5aaa4e17f8d2870d62. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.2:41589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:25.331336 812 raft_consensus.cc:2258] T 4d2cd9e84f3c46679226b515576d07f8 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.333359 812 tablet_replica.cc:330] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.333951 812 raft_consensus.cc:2229] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.334702 812 raft_consensus.cc:2258] T a034f41b715c4999a425f140bf2596d7 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.336647 812 tablet_replica.cc:330] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.337270 812 raft_consensus.cc:2229] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.338124 812 raft_consensus.cc:2258] T fdbc6827e58a491b85734200a08c259b P 36bc8e53f14a42319c2e8fcd5004ec6c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.339919 812 tablet_replica.cc:330] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.340579 812 raft_consensus.cc:2229] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:25.341356 812 raft_consensus.cc:2258] T 088ec2985f1e4c2bb6b5b475b3411865 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.343392 812 tablet_replica.cc:330] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c: stopping tablet replica
I20260924 00:21:25.343914 812 raft_consensus.cc:2229] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.344723 812 raft_consensus.cc:2258] T e0953f1c7d15425daf6292577384b885 P 36bc8e53f14a42319c2e8fcd5004ec6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.390666 812 tablet_server.cc:194] TabletServer@127.0.203.3:0 shutdown complete.
I20260924 00:21:25.410389 812 master.cc:537] Master@127.0.203.62:42997 shutting down...
I20260924 00:21:25.438112 812 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:21:25.438850 812 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:25.439518 812 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2e5227c91c0e4163a5da679ac1764175: stopping tablet replica
I20260924 00:21:25.460474 812 master.cc:552] Master@127.0.203.62:42997 shutdown complete.
1053ed314371460e9d47952fe809723c 5
36bc8e53f14a42319c2e8fcd5004ec6c 1
7fd889ee03954c5aaa4e17f8d2870d62 4
1053ed314371460e9d47952fe809723c 3
36bc8e53f14a42319c2e8fcd5004ec6c 3
7fd889ee03954c5aaa4e17f8d2870d62 4
1053ed314371460e9d47952fe809723c 8
36bc8e53f14a42319c2e8fcd5004ec6c 1
7fd889ee03954c5aaa4e17f8d2870d62 1
1053ed314371460e9d47952fe809723c 4
36bc8e53f14a42319c2e8fcd5004ec6c 3
7fd889ee03954c5aaa4e17f8d2870d62 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29497 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260924 00:21:25.508148 812 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.203.62:36337
I20260924 00:21:25.509434 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:21:25.516808 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:25.516858 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:25.517567 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:21:25.518420 812 server_base.cc:1034] running on GCE node
I20260924 00:21:25.519438 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:21:25.519659 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:21:25.519804 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209285519791 us; error 0 us; skew 500 ppm
I20260924 00:21:25.520309 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:21:25.537521 812 webserver.cc:460] Webserver started at http://127.0.203.62:34993/ using document root <none> and password file <none>
I20260924 00:21:25.538122 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:21:25.538365 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:21:25.538688 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:21:25.540040 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/master-0-root/instance:
uuid: "868c66955d2345848e02c62dafaf4a19"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.545271 812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260924 00:21:25.548976 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:25.549932 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:21:25.550174 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/master-0-root
uuid: "868c66955d2345848e02c62dafaf4a19"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.550437 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:21:25.564105 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:21:25.565411 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:21:25.609354 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.62:36337
I20260924 00:21:25.609434 1928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.62:36337 every 8 connection(s)
I20260924 00:21:25.613487 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:25.624598 1929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Bootstrap starting.
I20260924 00:21:25.629383 1929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:25.634032 1929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: No bootstrap required, opened a new log
I20260924 00:21:25.636227 1929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868c66955d2345848e02c62dafaf4a19" member_type: VOTER }
I20260924 00:21:25.636775 1929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:25.637024 1929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868c66955d2345848e02c62dafaf4a19, State: Initialized, Role: FOLLOWER
I20260924 00:21:25.637670 1929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868c66955d2345848e02c62dafaf4a19" member_type: VOTER }
I20260924 00:21:25.638140 1929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:21:25.638352 1929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:21:25.638586 1929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:25.643157 1929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868c66955d2345848e02c62dafaf4a19" member_type: VOTER }
I20260924 00:21:25.643677 1929 leader_election.cc:302] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 868c66955d2345848e02c62dafaf4a19; no voters:
I20260924 00:21:25.644793 1929 leader_election.cc:288] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:21:25.645346 1932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:25.646775 1932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [term 1 LEADER]: Becoming Leader. State: Replica: 868c66955d2345848e02c62dafaf4a19, State: Running, Role: LEADER
I20260924 00:21:25.647498 1932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868c66955d2345848e02c62dafaf4a19" member_type: VOTER }
I20260924 00:21:25.647678 1929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:21:25.654749 1934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 868c66955d2345848e02c62dafaf4a19. Latest consensus state: current_term: 1 leader_uuid: "868c66955d2345848e02c62dafaf4a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868c66955d2345848e02c62dafaf4a19" member_type: VOTER } }
I20260924 00:21:25.655578 1933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "868c66955d2345848e02c62dafaf4a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "868c66955d2345848e02c62dafaf4a19" member_type: VOTER } }
I20260924 00:21:25.656404 1934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:21:25.656545 1933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:21:25.664906 1941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:21:25.666771 812 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260924 00:21:25.672000 1941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:21:25.680950 1941 catalog_manager.cc:1347] Generated new cluster ID: 05da61cfd3d641bd86734b823370149f
I20260924 00:21:25.681272 1941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:21:25.709934 1941 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:21:25.712109 1941 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:21:25.723254 1941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Generated new TSK 0
I20260924 00:21:25.723976 1941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:21:25.733889 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:21:25.740783 1950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:25.741438 1951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:21:25.743328 812 server_base.cc:1034] running on GCE node
W20260924 00:21:25.743470 1953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:21:25.744630 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:21:25.744812 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:21:25.745012 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209285744982 us; error 0 us; skew 500 ppm
I20260924 00:21:25.745743 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:21:25.748414 812 webserver.cc:460] Webserver started at http://127.0.203.1:39237/ using document root <none> and password file <none>
I20260924 00:21:25.748976 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:21:25.749258 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:21:25.749543 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:21:25.750788 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/instance:
uuid: "3eb374c9744b4c7e92dc749aece177dd"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.755448 812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 00:21:25.759107 1958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:25.760015 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:21:25.760262 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root
uuid: "3eb374c9744b4c7e92dc749aece177dd"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.760510 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:21:25.782876 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:21:25.784196 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:21:25.785838 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:21:25.788326 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:21:25.788794 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:25.789141 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:21:25.789362 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 00:21:25.838217 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.1:41775
I20260924 00:21:25.838439 2020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.1:41775 every 8 connection(s)
I20260924 00:21:25.843231 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:21:25.851011 2025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:25.853475 2026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:25.855639 2028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:21:25.856909 812 server_base.cc:1034] running on GCE node
I20260924 00:21:25.858917 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:21:25.859206 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:21:25.859485 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209285859453 us; error 0 us; skew 500 ppm
I20260924 00:21:25.859556 2021 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36337
I20260924 00:21:25.860001 2021 heartbeater.cc:464] Registering TS with master...
I20260924 00:21:25.860234 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:21:25.860777 2021 heartbeater.cc:511] Master 127.0.203.62:36337 requested a full tablet report, sending...
I20260924 00:21:25.863514 1894 ts_manager.cc:194] Registered new tserver with Master: 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:25.863689 812 webserver.cc:460] Webserver started at http://127.0.203.2:43729/ using document root <none> and password file <none>
I20260924 00:21:25.864288 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:21:25.864512 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:21:25.864773 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:21:25.865891 1894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33232
I20260924 00:21:25.866014 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/instance:
uuid: "948356bf66a941ef8249ecbd0ec94f6d"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.871084 812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:21:25.874647 2033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:25.875527 812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 00:21:25.875927 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root
uuid: "948356bf66a941ef8249ecbd0ec94f6d"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.876334 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:21:25.900559 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:21:25.901917 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:21:25.903378 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:21:25.905783 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:21:25.906008 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:25.906256 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:21:25.906425 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:25.951009 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.2:39591
I20260924 00:21:25.951121 2095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.2:39591 every 8 connection(s)
I20260924 00:21:25.955936 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:21:25.963663 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:25.965818 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:25.970403 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:21:25.971455 2096 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36337
I20260924 00:21:25.971484 812 server_base.cc:1034] running on GCE node
I20260924 00:21:25.972007 2096 heartbeater.cc:464] Registering TS with master...
I20260924 00:21:25.972673 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:21:25.972887 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:21:25.972836 2096 heartbeater.cc:511] Master 127.0.203.62:36337 requested a full tablet report, sending...
I20260924 00:21:25.973280 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209285973262 us; error 0 us; skew 500 ppm
I20260924 00:21:25.973976 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:21:25.975597 1894 ts_manager.cc:194] Registered new tserver with Master: 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:25.977193 812 webserver.cc:460] Webserver started at http://127.0.203.3:34405/ using document root <none> and password file <none>
I20260924 00:21:25.977913 1894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33238
I20260924 00:21:25.978102 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:21:25.978449 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:21:25.978773 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:21:25.980051 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/instance:
uuid: "8c129c01e1c047628f6a87b001aa7258"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.985400 812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 00:21:25.988864 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:25.990033 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:21:25.990362 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root
uuid: "8c129c01e1c047628f6a87b001aa7258"
format_stamp: "Formatted at 2026-09-24 00:21:25 on dist-test-slave-r18p"
I20260924 00:21:25.990667 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:21:26.012512 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:21:26.013828 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:21:26.015326 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:21:26.017652 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:21:26.017886 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:26.018131 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:21:26.018294 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:26.060653 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.3:36385
I20260924 00:21:26.060961 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.3:36385 every 8 connection(s)
I20260924 00:21:26.079771 2170 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36337
I20260924 00:21:26.080305 2170 heartbeater.cc:464] Registering TS with master...
I20260924 00:21:26.081082 2170 heartbeater.cc:511] Master 127.0.203.62:36337 requested a full tablet report, sending...
I20260924 00:21:26.084105 1894 ts_manager.cc:194] Registered new tserver with Master: 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:26.084995 812 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020882421s
I20260924 00:21:26.086282 1894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33246
I20260924 00:21:26.087549 812 test_util.cc:268] Using random seed: -1802362009
I20260924 00:21:26.116209 1894 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33250:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:21:26.122579 1894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:21:26.373601 1986 tablet_service.cc:1467] Processing CreateTablet for tablet bf0af13699974118a86f602c1fa9490b (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260924 00:21:26.375473 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0af13699974118a86f602c1fa9490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.376547 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 20c9e32aa31c48a5b4d98f6813c8460a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260924 00:21:26.378163 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c9e32aa31c48a5b4d98f6813c8460a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.378352 2135 tablet_service.cc:1467] Processing CreateTablet for tablet bf0af13699974118a86f602c1fa9490b (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260924 00:21:26.379725 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0af13699974118a86f602c1fa9490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.379506 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 9700c37c6830431dbb61c9f7673b99bb (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260924 00:21:26.380777 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9700c37c6830431dbb61c9f7673b99bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.382066 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 313cbaaa6f5743a1b44a28217177fb4e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260924 00:21:26.382654 1980 tablet_service.cc:1467] Processing CreateTablet for tablet 93bfcd566e3e4c77b3a672d5705a546f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260924 00:21:26.383452 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313cbaaa6f5743a1b44a28217177fb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.384078 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bfcd566e3e4c77b3a672d5705a546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.385641 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 9700c37c6830431dbb61c9f7673b99bb (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260924 00:21:26.386332 2132 tablet_service.cc:1467] Processing CreateTablet for tablet 313cbaaa6f5743a1b44a28217177fb4e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260924 00:21:26.387082 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9700c37c6830431dbb61c9f7673b99bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.387673 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313cbaaa6f5743a1b44a28217177fb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.389591 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 6479ba76aef8450791098f4ac4fcee4e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260924 00:21:26.389933 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 150b87da8b7e456cb68c4549975535df (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260924 00:21:26.390353 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 970b194b81a34f3db37adc13d58b0a01 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260924 00:21:26.391108 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6479ba76aef8450791098f4ac4fcee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.391644 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970b194b81a34f3db37adc13d58b0a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.391885 1982 tablet_service.cc:1467] Processing CreateTablet for tablet 970b194b81a34f3db37adc13d58b0a01 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260924 00:21:26.393303 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970b194b81a34f3db37adc13d58b0a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.394357 2130 tablet_service.cc:1467] Processing CreateTablet for tablet 20c9e32aa31c48a5b4d98f6813c8460a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260924 00:21:26.395697 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c9e32aa31c48a5b4d98f6813c8460a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.387037 1979 tablet_service.cc:1467] Processing CreateTablet for tablet 88e267ffb8c4467b93815c1a9e3d3664 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260924 00:21:26.398620 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e267ffb8c4467b93815c1a9e3d3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.398296 1977 tablet_service.cc:1467] Processing CreateTablet for tablet 217f6c4b9f5c44d0b3856a8d131557cd (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260924 00:21:26.378352 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 150b87da8b7e456cb68c4549975535df (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260924 00:21:26.399773 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217f6c4b9f5c44d0b3856a8d131557cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.400594 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150b87da8b7e456cb68c4549975535df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.402853 2129 tablet_service.cc:1467] Processing CreateTablet for tablet 93bfcd566e3e4c77b3a672d5705a546f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260924 00:21:26.404318 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bfcd566e3e4c77b3a672d5705a546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.406414 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 88e267ffb8c4467b93815c1a9e3d3664 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260924 00:21:26.407882 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e267ffb8c4467b93815c1a9e3d3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.410300 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150b87da8b7e456cb68c4549975535df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.411366 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 6479ba76aef8450791098f4ac4fcee4e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260924 00:21:26.429579 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6479ba76aef8450791098f4ac4fcee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.446610 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 217f6c4b9f5c44d0b3856a8d131557cd (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260924 00:21:26.448082 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217f6c4b9f5c44d0b3856a8d131557cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.476670 1980 tablet_service.cc:1467] Processing CreateTablet for tablet 66641de0607c476a958313ea66dff4db (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260924 00:21:26.479526 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66641de0607c476a958313ea66dff4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.482697 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff940e15e1849efb194c908520c212f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260924 00:21:26.484107 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff940e15e1849efb194c908520c212f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.487066 1981 tablet_service.cc:1467] Processing CreateTablet for tablet f9aa1fbdb2cf48bb9bdb4c51026f3c83 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260924 00:21:26.491750 2185 tablet_bootstrap.cc:492] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:26.493898 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 66641de0607c476a958313ea66dff4db (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260924 00:21:26.495301 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66641de0607c476a958313ea66dff4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.498129 2185 tablet_bootstrap.cc:654] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:26.508670 2133 tablet_service.cc:1467] Processing CreateTablet for tablet f9aa1fbdb2cf48bb9bdb4c51026f3c83 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260924 00:21:26.510135 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9aa1fbdb2cf48bb9bdb4c51026f3c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.529879 2133 tablet_service.cc:1467] Processing CreateTablet for tablet d6c2f84c42b9478e954135118d17b4e2 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260924 00:21:26.531373 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c2f84c42b9478e954135118d17b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.533960 2061 tablet_service.cc:1467] Processing CreateTablet for tablet bf0af13699974118a86f602c1fa9490b (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260924 00:21:26.535373 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0af13699974118a86f602c1fa9490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.535118 2132 tablet_service.cc:1467] Processing CreateTablet for tablet bd66a46f82114470b22385448fd57d51 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260924 00:21:26.536425 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd66a46f82114470b22385448fd57d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.542619 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 150b87da8b7e456cb68c4549975535df (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260924 00:21:26.544051 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150b87da8b7e456cb68c4549975535df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.543706 1979 tablet_service.cc:1467] Processing CreateTablet for tablet d6c2f84c42b9478e954135118d17b4e2 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260924 00:21:26.544947 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c2f84c42b9478e954135118d17b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.547878 2133 tablet_service.cc:1467] Processing CreateTablet for tablet ed845c50caec4187b2daed75db23bfd8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260924 00:21:26.549379 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed845c50caec4187b2daed75db23bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.550807 1982 tablet_service.cc:1467] Processing CreateTablet for tablet bd66a46f82114470b22385448fd57d51 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260924 00:21:26.552237 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd66a46f82114470b22385448fd57d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.561116 1977 tablet_service.cc:1467] Processing CreateTablet for tablet ed845c50caec4187b2daed75db23bfd8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260924 00:21:26.562836 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed845c50caec4187b2daed75db23bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.565212 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 9700c37c6830431dbb61c9f7673b99bb (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260924 00:21:26.565668 2058 tablet_service.cc:1467] Processing CreateTablet for tablet 313cbaaa6f5743a1b44a28217177fb4e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260924 00:21:26.567035 2057 tablet_service.cc:1467] Processing CreateTablet for tablet 970b194b81a34f3db37adc13d58b0a01 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260924 00:21:26.568202 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260924 00:21:26.586151 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313cbaaa6f5743a1b44a28217177fb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.569175 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260924 00:21:26.586916 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970b194b81a34f3db37adc13d58b0a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.587534 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.589756 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 8e92f7a727fb4c1da7807d1429d98861 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260924 00:21:26.591257 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e92f7a727fb4c1da7807d1429d98861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.569489 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 217f6c4b9f5c44d0b3856a8d131557cd (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260924 00:21:26.569689 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 88e267ffb8c4467b93815c1a9e3d3664 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260924 00:21:26.571462 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 6479ba76aef8450791098f4ac4fcee4e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260924 00:21:26.573113 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 80ac2f44b186407ba12e93016e1ae270 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260924 00:21:26.573886 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 93bfcd566e3e4c77b3a672d5705a546f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260924 00:21:26.593077 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9aa1fbdb2cf48bb9bdb4c51026f3c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.574896 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 20c9e32aa31c48a5b4d98f6813c8460a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260924 00:21:26.575145 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 80ac2f44b186407ba12e93016e1ae270 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260924 00:21:26.594414 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ac2f44b186407ba12e93016e1ae270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.595775 2129 tablet_service.cc:1467] Processing CreateTablet for tablet efa7fce6500d4a22b34baf31a7dbb247 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260924 00:21:26.597514 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.598439 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e267ffb8c4467b93815c1a9e3d3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.600584 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6479ba76aef8450791098f4ac4fcee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.575649 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4f367683e243cdb37181c1bdb546ec (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260924 00:21:26.601974 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4f367683e243cdb37181c1bdb546ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.605831 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bfcd566e3e4c77b3a672d5705a546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.629338 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7fce6500d4a22b34baf31a7dbb247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.631488 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c9e32aa31c48a5b4d98f6813c8460a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.584537 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4f367683e243cdb37181c1bdb546ec (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260924 00:21:26.641184 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ac2f44b186407ba12e93016e1ae270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.641831 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217f6c4b9f5c44d0b3856a8d131557cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.645744 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9700c37c6830431dbb61c9f7673b99bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.658628 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 2a03e76a8d83458ca79bbaef1eee19cb (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260924 00:21:26.660799 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a03e76a8d83458ca79bbaef1eee19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.663233 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4f367683e243cdb37181c1bdb546ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 00:21:26.660892 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:21:26.664317 2130 tablet_service.cc:1467] Processing CreateTablet for tablet 54b1b238378448a5987c49aafc3b6313 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260924 00:21:26.667354 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b1b238378448a5987c49aafc3b6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.666065 1985 tablet_service.cc:1467] Processing CreateTablet for tablet efa7fce6500d4a22b34baf31a7dbb247 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260924 00:21:26.674111 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7fce6500d4a22b34baf31a7dbb247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.678336 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 8e92f7a727fb4c1da7807d1429d98861 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260924 00:21:26.679760 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e92f7a727fb4c1da7807d1429d98861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.515166 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff940e15e1849efb194c908520c212f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260924 00:21:26.694085 2188 tablet_bootstrap.cc:492] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:26.696828 2185 tablet_bootstrap.cc:492] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:26.697525 2185 ts_tablet_manager.cc:1397] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.206s user 0.034s sys 0.031s
I20260924 00:21:26.700958 2185 raft_consensus.cc:348] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:26.702565 2185 raft_consensus.cc:374] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:26.703500 2185 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:26.704708 2185 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:26.706668 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 66641de0607c476a958313ea66dff4db (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260924 00:21:26.708277 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66641de0607c476a958313ea66dff4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.721743 2186 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:26.730782 2129 tablet_service.cc:1467] Processing CreateTablet for tablet 708537d593c14817accda67f9fd081b4 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260924 00:21:26.734395 2060 tablet_service.cc:1467] Processing CreateTablet for tablet f9aa1fbdb2cf48bb9bdb4c51026f3c83 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260924 00:21:26.735814 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9aa1fbdb2cf48bb9bdb4c51026f3c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.735555 2128 tablet_service.cc:1467] Processing CreateTablet for tablet ba683df200644d888896fb84dbc748e8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260924 00:21:26.738093 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 2a03e76a8d83458ca79bbaef1eee19cb (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260924 00:21:26.739856 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a03e76a8d83458ca79bbaef1eee19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.742753 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708537d593c14817accda67f9fd081b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.749473 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba683df200644d888896fb84dbc748e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.754951 2186 tablet_bootstrap.cc:654] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:26.755574 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff940e15e1849efb194c908520c212f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.780920 2135 tablet_service.cc:1467] Processing CreateTablet for tablet cfca69df981d4ab89f1ea62a1d8c1557 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260924 00:21:26.786728 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfca69df981d4ab89f1ea62a1d8c1557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.763974 2057 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff940e15e1849efb194c908520c212f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260924 00:21:26.787859 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff940e15e1849efb194c908520c212f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.790132 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 708537d593c14817accda67f9fd081b4 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260924 00:21:26.797250 2056 tablet_service.cc:1467] Processing CreateTablet for tablet d6c2f84c42b9478e954135118d17b4e2 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260924 00:21:26.798588 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c2f84c42b9478e954135118d17b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.772524 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 54b1b238378448a5987c49aafc3b6313 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260924 00:21:26.803766 2058 tablet_service.cc:1467] Processing CreateTablet for tablet bd66a46f82114470b22385448fd57d51 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260924 00:21:26.804175 2185 ts_tablet_manager.cc:1428] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.106s user 0.016s sys 0.000s
I20260924 00:21:26.807873 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd66a46f82114470b22385448fd57d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.810516 2054 tablet_service.cc:1467] Processing CreateTablet for tablet ed845c50caec4187b2daed75db23bfd8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260924 00:21:26.811848 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed845c50caec4187b2daed75db23bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.820704 1984 tablet_service.cc:1467] Processing CreateTablet for tablet cfca69df981d4ab89f1ea62a1d8c1557 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260924 00:21:26.821382 1977 tablet_service.cc:1467] Processing CreateTablet for tablet f2f86eadee0b4bbcbfd3437d30d304b5 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260924 00:21:26.822151 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfca69df981d4ab89f1ea62a1d8c1557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.824002 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1695e8b6d44e39a87ae6431db2eb9b (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260924 00:21:26.833683 2185 tablet_bootstrap.cc:492] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:26.834371 2021 heartbeater.cc:503] Master 127.0.203.62:36337 was elected leader, sending a full tablet report...
I20260924 00:21:26.828444 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 80ac2f44b186407ba12e93016e1ae270 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260924 00:21:26.820704 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 86edca4e63224252bc39bdd9c0bddca6 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260924 00:21:26.838564 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86edca4e63224252bc39bdd9c0bddca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.839412 2053 tablet_service.cc:1467] Processing CreateTablet for tablet efa7fce6500d4a22b34baf31a7dbb247 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260924 00:21:26.837747 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ac2f44b186407ba12e93016e1ae270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.854956 1985 tablet_service.cc:1467] Processing CreateTablet for tablet f2730c02ac0a4812841172254e284b90 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260924 00:21:26.856992 1982 tablet_service.cc:1467] Processing CreateTablet for tablet ba683df200644d888896fb84dbc748e8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260924 00:21:26.857394 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260924 00:21:26.861732 2188 tablet_bootstrap.cc:654] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:26.859831 1980 tablet_service.cc:1467] Processing CreateTablet for tablet 2f75d53c3ccf437cac00ba5d83b2c165 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260924 00:21:26.865568 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1695e8b6d44e39a87ae6431db2eb9b (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260924 00:21:26.866153 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7fce6500d4a22b34baf31a7dbb247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.867475 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1695e8b6d44e39a87ae6431db2eb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.868324 1979 tablet_service.cc:1467] Processing CreateTablet for tablet 86edca4e63224252bc39bdd9c0bddca6 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260924 00:21:26.872087 2130 tablet_service.cc:1467] Processing CreateTablet for tablet f2730c02ac0a4812841172254e284b90 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260924 00:21:26.873732 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4f367683e243cdb37181c1bdb546ec (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260924 00:21:26.847278 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708537d593c14817accda67f9fd081b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.817034 2126 tablet_service.cc:1467] Processing CreateTablet for tablet f2f86eadee0b4bbcbfd3437d30d304b5 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260924 00:21:26.875376 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4f367683e243cdb37181c1bdb546ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.886690 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.897835 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f75d53c3ccf437cac00ba5d83b2c165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.897464 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 2f75d53c3ccf437cac00ba5d83b2c165 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260924 00:21:26.898512 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f86eadee0b4bbcbfd3437d30d304b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.899159 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86edca4e63224252bc39bdd9c0bddca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.900701 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1695e8b6d44e39a87ae6431db2eb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.901321 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b1b238378448a5987c49aafc3b6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.897502 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2730c02ac0a4812841172254e284b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.907629 2185 tablet_bootstrap.cc:654] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:26.909192 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba683df200644d888896fb84dbc748e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.909425 2189 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:26.910120 2189 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:26.912635 2189 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:26.917501 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2730c02ac0a4812841172254e284b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.943470 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 91be317234ca451fad76824aef0861a6 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260924 00:21:26.928515 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 8e92f7a727fb4c1da7807d1429d98861 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260924 00:21:26.931110 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f75d53c3ccf437cac00ba5d83b2c165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.926321 2132 tablet_service.cc:1467] Processing CreateTablet for tablet aa5e2f99029440bd9d42ec2a01200d4a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260924 00:21:26.969962 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91be317234ca451fad76824aef0861a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.957194 2186 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:26.981478 2096 heartbeater.cc:503] Master 127.0.203.62:36337 was elected leader, sending a full tablet report...
I20260924 00:21:26.957054 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e92f7a727fb4c1da7807d1429d98861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:26.997531 2186 ts_tablet_manager.cc:1397] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.276s user 0.020s sys 0.003s
I20260924 00:21:27.000212 2186 raft_consensus.cc:348] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.008365 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 447705637cdc44668a870a52a7a3871a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260924 00:21:27.000964 2186 raft_consensus.cc:374] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.014743 2186 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.015237 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 2a03e76a8d83458ca79bbaef1eee19cb (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260924 00:21:27.016039 2185 tablet_bootstrap.cc:492] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:27.015620 2186 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.016644 2185 ts_tablet_manager.cc:1397] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.183s user 0.016s sys 0.004s
I20260924 00:21:27.019966 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f86eadee0b4bbcbfd3437d30d304b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.025494 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5e2f99029440bd9d42ec2a01200d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.030906 2185 raft_consensus.cc:348] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.031819 2185 raft_consensus.cc:374] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.032173 2185 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.032895 2185 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.041580 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447705637cdc44668a870a52a7a3871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.042308 1982 tablet_service.cc:1467] Processing CreateTablet for tablet 91be317234ca451fad76824aef0861a6 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260924 00:21:27.044646 2185 ts_tablet_manager.cc:1428] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20260924 00:21:27.046034 2185 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:27.046028 1977 tablet_service.cc:1467] Processing CreateTablet for tablet aa5e2f99029440bd9d42ec2a01200d4a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260924 00:21:27.049357 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5e2f99029440bd9d42ec2a01200d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.050540 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 24240f07434f49e9bef897dac3fbe801 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260924 00:21:27.051259 1979 tablet_service.cc:1467] Processing CreateTablet for tablet 24240f07434f49e9bef897dac3fbe801 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260924 00:21:27.052032 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24240f07434f49e9bef897dac3fbe801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.053973 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 447705637cdc44668a870a52a7a3871a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260924 00:21:27.052131 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91be317234ca451fad76824aef0861a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.055837 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447705637cdc44668a870a52a7a3871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.058367 1984 tablet_service.cc:1467] Processing CreateTablet for tablet f63d709a34bd44a1b37eee2f1eb2112c (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260924 00:21:27.059350 2057 tablet_service.cc:1467] Processing CreateTablet for tablet 54b1b238378448a5987c49aafc3b6313 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260924 00:21:27.061049 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b1b238378448a5987c49aafc3b6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.061466 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24240f07434f49e9bef897dac3fbe801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.068226 2061 tablet_service.cc:1467] Processing CreateTablet for tablet ba683df200644d888896fb84dbc748e8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260924 00:21:27.092746 2186 ts_tablet_manager.cc:1428] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.095s user 0.005s sys 0.006s
I20260924 00:21:27.093784 2186 tablet_bootstrap.cc:492] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:27.094213 2129 tablet_service.cc:1467] Processing CreateTablet for tablet 55201d7dbea64daab55bc0144cc26c13 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260924 00:21:27.095497 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55201d7dbea64daab55bc0144cc26c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.070518 2054 tablet_service.cc:1467] Processing CreateTablet for tablet cfca69df981d4ab89f1ea62a1d8c1557 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260924 00:21:27.077454 2131 tablet_service.cc:1467] Processing CreateTablet for tablet f63d709a34bd44a1b37eee2f1eb2112c (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260924 00:21:27.068209 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 708537d593c14817accda67f9fd081b4 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260924 00:21:27.103403 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d709a34bd44a1b37eee2f1eb2112c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.103248 1986 tablet_service.cc:1467] Processing CreateTablet for tablet b76aee4623d7484e8ff6c1f1a13f5fea (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260924 00:21:27.104601 2196 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:27.105449 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708537d593c14817accda67f9fd081b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.105098 2196 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.087836 2185 tablet_bootstrap.cc:654] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.089329 2170 heartbeater.cc:503] Master 127.0.203.62:36337 was elected leader, sending a full tablet report...
I20260924 00:21:27.101075 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d709a34bd44a1b37eee2f1eb2112c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.101310 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1695e8b6d44e39a87ae6431db2eb9b (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260924 00:21:27.081485 2059 tablet_service.cc:1467] Processing CreateTablet for tablet f2f86eadee0b4bbcbfd3437d30d304b5 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260924 00:21:27.123257 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 55201d7dbea64daab55bc0144cc26c13 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260924 00:21:27.128209 2058 tablet_service.cc:1467] Processing CreateTablet for tablet 2f75d53c3ccf437cac00ba5d83b2c165 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260924 00:21:27.133271 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a03e76a8d83458ca79bbaef1eee19cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.133598 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 466d562826384d688b157b8830cfd5d1 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260924 00:21:27.134234 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f86eadee0b4bbcbfd3437d30d304b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.135097 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfca69df981d4ab89f1ea62a1d8c1557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.148722 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 466d562826384d688b157b8830cfd5d1 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260924 00:21:27.155660 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4607ccc19a497b8082abfda6bcc0b9 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260924 00:21:27.156780 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4607ccc19a497b8082abfda6bcc0b9 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260924 00:21:27.148743 2132 tablet_service.cc:1467] Processing CreateTablet for tablet b76aee4623d7484e8ff6c1f1a13f5fea (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260924 00:21:27.163029 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 86edca4e63224252bc39bdd9c0bddca6 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260924 00:21:27.167521 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55201d7dbea64daab55bc0144cc26c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.167150 2188 tablet_bootstrap.cc:492] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:27.110360 2196 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:27.169282 2188 ts_tablet_manager.cc:1397] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.476s user 0.012s sys 0.006s
I20260924 00:21:27.170161 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76aee4623d7484e8ff6c1f1a13f5fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.178375 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466d562826384d688b157b8830cfd5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.171083 1980 tablet_service.cc:1467] Processing CreateTablet for tablet 85bede790e49439d97e8031810eee902 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260924 00:21:27.179847 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bede790e49439d97e8031810eee902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.181630 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466d562826384d688b157b8830cfd5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.185614 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4607ccc19a497b8082abfda6bcc0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.181560 1977 tablet_service.cc:1467] Processing CreateTablet for tablet 771618125e684526a907a103e7694ae8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260924 00:21:27.190330 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771618125e684526a907a103e7694ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.190809 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f75d53c3ccf437cac00ba5d83b2c165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.192481 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76aee4623d7484e8ff6c1f1a13f5fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.194747 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 771618125e684526a907a103e7694ae8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260924 00:21:27.196237 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771618125e684526a907a103e7694ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.203284 2130 tablet_service.cc:1467] Processing CreateTablet for tablet c4982c422b5247e29807c869b18b7eb8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260924 00:21:27.203552 2188 raft_consensus.cc:348] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.204361 2188 raft_consensus.cc:374] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.204664 2188 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.204809 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4982c422b5247e29807c869b18b7eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.209528 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:27.211905 1960 leader_election.cc:341] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.173530 2186 tablet_bootstrap.cc:654] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.218055 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 13ff368f645e4ee7acdef88492f25ae7 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260924 00:21:27.220268 2057 tablet_service.cc:1467] Processing CreateTablet for tablet f2730c02ac0a4812841172254e284b90 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260924 00:21:27.236238 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86edca4e63224252bc39bdd9c0bddca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.241827 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba683df200644d888896fb84dbc748e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.236428 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4607ccc19a497b8082abfda6bcc0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.242760 1979 tablet_service.cc:1467] Processing CreateTablet for tablet c4982c422b5247e29807c869b18b7eb8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260924 00:21:27.233501 2188 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.247897 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1695e8b6d44e39a87ae6431db2eb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.245043 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4982c422b5247e29807c869b18b7eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.261655 2190 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:27.262214 2190 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.262866 2185 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:27.263437 2185 ts_tablet_manager.cc:1397] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.218s user 0.010s sys 0.011s
I20260924 00:21:27.264612 2190 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:27.259173 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 85bede790e49439d97e8031810eee902 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260924 00:21:27.266933 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:27.267633 2185 raft_consensus.cc:348] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.268556 2185 raft_consensus.cc:374] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.268893 2185 raft_consensus.cc:729] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
W20260924 00:21:27.269728 1960 leader_election.cc:341] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.273692 2185 consensus_queue.cc:260] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.279014 2185 ts_tablet_manager.cc:1428] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.015s user 0.008s sys 0.002s
I20260924 00:21:27.280120 2185 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:27.286729 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2730c02ac0a4812841172254e284b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.287931 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ff368f645e4ee7acdef88492f25ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.289538 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bede790e49439d97e8031810eee902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.255048 2053 tablet_service.cc:1467] Processing CreateTablet for tablet aa5e2f99029440bd9d42ec2a01200d4a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260924 00:21:27.290813 2190 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:27.291189 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5e2f99029440bd9d42ec2a01200d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.291286 2190 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.289242 2131 tablet_service.cc:1467] Processing CreateTablet for tablet d581036c5b6e448cbae056fa4fc15f78 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260924 00:21:27.304188 2188 ts_tablet_manager.cc:1428] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.134s user 0.009s sys 0.008s
I20260924 00:21:27.306707 1982 tablet_service.cc:1467] Processing CreateTablet for tablet 13ff368f645e4ee7acdef88492f25ae7 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260924 00:21:27.309819 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d581036c5b6e448cbae056fa4fc15f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.313169 2190 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:27.314791 2188 tablet_bootstrap.cc:492] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:27.314668 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:27.315960 1960 leader_election.cc:341] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.326927 2185 tablet_bootstrap.cc:654] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.333727 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ff368f645e4ee7acdef88492f25ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.340515 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:27.342308 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:27.343930 1960 leader_election.cc:341] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.344383 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:27.345523 2190 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:27.324730 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 91be317234ca451fad76824aef0861a6 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
W20260924 00:21:27.349525 1960 leader_election.cc:341] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.363636 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:27.365175 2129 tablet_service.cc:1467] Processing CreateTablet for tablet 95ac9e39e8934ecdb84ac79399dcbd9c (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260924 00:21:27.351162 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260924 00:21:27.374573 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.375710 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:27.376039 2186 tablet_bootstrap.cc:492] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:27.376590 2186 ts_tablet_manager.cc:1397] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.283s user 0.004s sys 0.011s
W20260924 00:21:27.377374 2109 leader_election.cc:341] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.379726 2186 raft_consensus.cc:348] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:27.380697 2186 raft_consensus.cc:374] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.352034 1978 tablet_service.cc:1467] Processing CreateTablet for tablet d581036c5b6e448cbae056fa4fc15f78 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260924 00:21:27.381883 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d581036c5b6e448cbae056fa4fc15f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.383523 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 25490e5b518e4abf90527c9e0dd7723e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260924 00:21:27.384789 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25490e5b518e4abf90527c9e0dd7723e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.353701 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260924 00:21:27.386042 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.368966 2132 tablet_service.cc:1467] Processing CreateTablet for tablet a4917c5d11b34f7aaa5cbdcf98709f22 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260924 00:21:27.390121 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:27.360934 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 24240f07434f49e9bef897dac3fbe801 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260924 00:21:27.367285 2190 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:27.349622 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:27.369897 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91be317234ca451fad76824aef0861a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.370079 2188 tablet_bootstrap.cc:654] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.381002 2186 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.386574 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 447705637cdc44668a870a52a7a3871a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260924 00:21:27.360453 1980 tablet_service.cc:1467] Processing CreateTablet for tablet a4917c5d11b34f7aaa5cbdcf98709f22 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20260924 00:21:27.392192 2110 leader_election.cc:341] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.388633 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4917c5d11b34f7aaa5cbdcf98709f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.400547 2186 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:27.402930 2186 ts_tablet_manager.cc:1428] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.026s user 0.009s sys 0.000s
I20260924 00:21:27.403856 2186 tablet_bootstrap.cc:492] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:27.373225 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 95ac9e39e8934ecdb84ac79399dcbd9c (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260924 00:21:27.405226 2110 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:27.410406 2196 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 00:21:27.411257 1960 leader_election.cc:341] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.411777 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:27.396859 2058 tablet_service.cc:1467] Processing CreateTablet for tablet 466d562826384d688b157b8830cfd5d1 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260924 00:21:27.412883 2134 tablet_service.cc:1467] Processing CreateTablet for tablet aaea92f011f94575ab225f58d2a0db06 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260924 00:21:27.417829 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24240f07434f49e9bef897dac3fbe801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.418190 2131 tablet_service.cc:1467] Processing CreateTablet for tablet e4a9343576d1478caf59e9f5629ce211 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260924 00:21:27.413116 2190 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:27.430344 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa219f0283c4632b6601e60980ae25a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260924 00:21:27.430931 2054 tablet_service.cc:1467] Processing CreateTablet for tablet b76aee4623d7484e8ff6c1f1a13f5fea (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260924 00:21:27.434662 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447705637cdc44668a870a52a7a3871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.439711 2052 tablet_service.cc:1467] Processing CreateTablet for tablet f63d709a34bd44a1b37eee2f1eb2112c (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260924 00:21:27.395542 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 55201d7dbea64daab55bc0144cc26c13 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260924 00:21:27.441339 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ac9e39e8934ecdb84ac79399dcbd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.442515 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a9343576d1478caf59e9f5629ce211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.443706 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaea92f011f94575ab225f58d2a0db06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.447261 2127 tablet_service.cc:1467] Processing CreateTablet for tablet c616f076fa3a4e75b7d3c51e60553d74 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260924 00:21:27.448740 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c616f076fa3a4e75b7d3c51e60553d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.450666 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa219f0283c4632b6601e60980ae25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.458664 2202 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:27.460472 2202 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.463303 2202 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:27.468076 1981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 36) took 1011 ms. Trace:
I20260924 00:21:27.479063 1977 tablet_service.cc:1467] Processing CreateTablet for tablet 25490e5b518e4abf90527c9e0dd7723e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260924 00:21:27.481416 1986 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 38) took 1004 ms. Trace:
I20260924 00:21:27.482051 1986 rpcz_store.cc:276] 0924 00:21:26.465486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.102390 (+636904us) service_pool.cc:227] Handling call
0924 00:21:27.104086 (+ 1696us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.104178 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.264068 (+159890us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.438686 (+174618us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.470087 (+ 31401us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":373}
I20260924 00:21:27.475172 1981 rpcz_store.cc:276] 0924 00:21:26.455256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.084155 (+628899us) service_pool.cc:227] Handling call
0924 00:21:27.124194 (+ 40039us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.124321 (+ 127us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.282269 (+157948us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.373419 (+ 91150us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.417635 (+ 44216us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5ce82 00007fb6daa758b9 00007fb6d99885d8 00007fb6d9a4c063 00007fb6d9a47e2f 00007fb6d9a485ed 00007fb6d99f9869 00007fb6d2ffbc44 00007fb6d2ffbbd2
0924 00:21:27.466576 (+ 48941us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3273,"spinlock_wait_cycles":30861568}
I20260924 00:21:27.485486 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76aee4623d7484e8ff6c1f1a13f5fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.487221 1981 tablet_service.cc:1467] Processing CreateTablet for tablet e4a9343576d1478caf59e9f5629ce211 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260924 00:21:27.489060 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4917c5d11b34f7aaa5cbdcf98709f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.491842 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ac9e39e8934ecdb84ac79399dcbd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.494160 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466d562826384d688b157b8830cfd5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.494675 1983 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 37) took 1033 ms. Trace:
I20260924 00:21:27.495419 2185 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:27.495913 2185 ts_tablet_manager.cc:1397] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.216s user 0.011s sys 0.008s
I20260924 00:21:27.498420 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4607ccc19a497b8082abfda6bcc0b9 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260924 00:21:27.498627 2185 raft_consensus.cc:348] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.499359 2185 raft_consensus.cc:374] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.499671 2185 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.499825 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55201d7dbea64daab55bc0144cc26c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.500373 2185 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.501765 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25490e5b518e4abf90527c9e0dd7723e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.502116 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4607ccc19a497b8082abfda6bcc0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.503079 2185 ts_tablet_manager.cc:1428] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260924 00:21:27.504040 2185 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:27.504791 1979 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa219f0283c4632b6601e60980ae25a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260924 00:21:27.506275 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa219f0283c4632b6601e60980ae25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.510027 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d709a34bd44a1b37eee2f1eb2112c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.527163 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a9343576d1478caf59e9f5629ce211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.533527 2186 tablet_bootstrap.cc:654] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.495077 1983 rpcz_store.cc:276] 0924 00:21:26.461618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.088522 (+626904us) service_pool.cc:227] Handling call
0924 00:21:27.134551 (+ 46029us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.134643 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.322288 (+187645us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.398640 (+ 76352us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.494319 (+ 95679us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5ce82 00007fb6daa758b9 00007fb6d99885d8 00007fb6d9a4c063 00007fb6d9a47e2f 00007fb6d9a485ed 00007fb6d99f9869 00007fb6d2ffbc44 00007fb6d2ffbbd2
0924 00:21:27.494614 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2995968}
I20260924 00:21:27.545831 1983 tablet_service.cc:1467] Processing CreateTablet for tablet c616f076fa3a4e75b7d3c51e60553d74 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260924 00:21:27.548421 1986 tablet_service.cc:1467] Processing CreateTablet for tablet aaea92f011f94575ab225f58d2a0db06 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260924 00:21:27.552337 2057 tablet_service.cc:1467] Processing CreateTablet for tablet 771618125e684526a907a103e7694ae8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260924 00:21:27.553388 2132 tablet_service.cc:1467] Processing CreateTablet for tablet e7be2c535b984fd6bf122044c990f72e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260924 00:21:27.585361 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7be2c535b984fd6bf122044c990f72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.585675 2188 tablet_bootstrap.cc:492] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:27.586346 2188 ts_tablet_manager.cc:1397] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.272s user 0.029s sys 0.017s
I20260924 00:21:27.594787 2185 tablet_bootstrap.cc:654] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:27.597496 1880 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 48118.4us
I20260924 00:21:27.574795 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771618125e684526a907a103e7694ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.618001 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 85bede790e49439d97e8031810eee902 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260924 00:21:27.621213 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c616f076fa3a4e75b7d3c51e60553d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.623540 2128 tablet_service.cc:1467] Processing CreateTablet for tablet d2ef9fa9e5dd48218a5c243de0c0e49a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260924 00:21:27.624974 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ef9fa9e5dd48218a5c243de0c0e49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.627065 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:27.628659 2035 leader_election.cc:341] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.629879 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bede790e49439d97e8031810eee902. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 00:21:27.634078 1879 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 82345.1us
I20260924 00:21:27.633819 2134 tablet_service.cc:1467] Processing CreateTablet for tablet d1caddf844ef49c0b3664f36bb76b869 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260924 00:21:27.635653 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1caddf844ef49c0b3664f36bb76b869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.607446 2135 tablet_service.cc:1467] Processing CreateTablet for tablet fc79ee9a960d47c2b7d3150a8aa37983 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260924 00:21:27.638775 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc79ee9a960d47c2b7d3150a8aa37983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.589121 2188 raft_consensus.cc:348] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:27.640956 2188 raft_consensus.cc:374] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.641526 2188 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.645206 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaea92f011f94575ab225f58d2a0db06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.646802 1982 tablet_service.cc:1467] Processing CreateTablet for tablet e7be2c535b984fd6bf122044c990f72e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260924 00:21:27.655846 2053 tablet_service.cc:1467] Processing CreateTablet for tablet c4982c422b5247e29807c869b18b7eb8 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260924 00:21:27.657817 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4982c422b5247e29807c869b18b7eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.633049 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 13ff368f645e4ee7acdef88492f25ae7 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260924 00:21:27.662838 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 832ee611b00d4a42b3245beb345af7ad (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260924 00:21:27.664266 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ff368f645e4ee7acdef88492f25ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.664541 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832ee611b00d4a42b3245beb345af7ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 00:21:27.704896 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260924 00:21:27.700340 1880 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 59763.5us
I20260924 00:21:27.698851 2130 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57740 (request call id 42) took 1056 ms. Trace:
I20260924 00:21:27.718930 1978 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 44) took 1009 ms. Trace:
I20260924 00:21:27.719383 1978 rpcz_store.cc:276] 0924 00:21:26.710018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.351071 (+641053us) service_pool.cc:227] Handling call
0924 00:21:27.381438 (+ 30367us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.381542 (+ 104us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.436916 (+ 55374us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.704406 (+267490us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.718862 (+ 14456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.716683 2130 rpcz_store.cc:276] 0924 00:21:26.641429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.202140 (+560711us) service_pool.cc:227] Handling call
0924 00:21:27.204246 (+ 2106us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.204344 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.447842 (+243498us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.632671 (+184829us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.694332 (+ 61661us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.723564 2130 tablet_service.cc:1467] Processing CreateTablet for tablet 92bd85e8dadb4876827acd9ff4be377f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260924 00:21:27.726900 2186 tablet_bootstrap.cc:492] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:27.717847 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7be2c535b984fd6bf122044c990f72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.730724 2185 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:27.731242 2185 ts_tablet_manager.cc:1397] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.227s user 0.015s sys 0.035s
I20260924 00:21:27.731253 1979 tablet_service.cc:1467] Processing CreateTablet for tablet d1caddf844ef49c0b3664f36bb76b869 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260924 00:21:27.732803 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1caddf844ef49c0b3664f36bb76b869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.733631 1978 tablet_service.cc:1467] Processing CreateTablet for tablet fc79ee9a960d47c2b7d3150a8aa37983 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260924 00:21:27.734925 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc79ee9a960d47c2b7d3150a8aa37983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.727442 2186 ts_tablet_manager.cc:1397] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.324s user 0.020s sys 0.015s
I20260924 00:21:27.724941 2052 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 35) took 1025 ms. Trace:
I20260924 00:21:27.749464 2052 rpcz_store.cc:276] 0924 00:21:26.699513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.392464 (+692951us) service_pool.cc:227] Handling call
0924 00:21:27.493492 (+101028us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b580008b898. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5cb7c 00007fb6d1b5c73b 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a6ad21 00007fb6d9a481e3 00007fb6d99f9869 00007fb6d2ffbc44 00007fb6d2ffbbd2 00007fb6d2ffbb01
0924 00:21:27.509376 (+ 15884us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.509472 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.625086 (+115614us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.724410 (+ 99324us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.724880 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18567296}
I20260924 00:21:27.759954 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:21:27.762863 2036 leader_election.cc:341] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.766153 2036 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:27.767041 2202 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:27.642473 2188 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:27.771674 2185 raft_consensus.cc:348] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.772462 2185 raft_consensus.cc:374] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.772739 2185 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.725080 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd85e8dadb4876827acd9ff4be377f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.778465 2188 ts_tablet_manager.cc:1428] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.192s user 0.007s sys 0.001s
I20260924 00:21:27.779525 2188 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:27.786803 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 832ee611b00d4a42b3245beb345af7ad (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260924 00:21:27.788754 1980 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 47) took 1025 ms. Trace:
I20260924 00:21:27.794612 2060 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 32) took 1097 ms. Trace:
I20260924 00:21:27.817915 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832ee611b00d4a42b3245beb345af7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.818118 1985 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 46) took 1073 ms. Trace:
I20260924 00:21:27.818149 2060 rpcz_store.cc:276] 0924 00:21:26.697511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.323775 (+626264us) service_pool.cc:227] Handling call
0924 00:21:27.369382 (+ 45607us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.369475 (+ 93us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.719628 (+350153us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.794073 (+ 74445us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.794509 (+ 436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.818569 1985 rpcz_store.cc:276] 0924 00:21:26.744467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.357503 (+613036us) service_pool.cc:227] Handling call
0924 00:21:27.425575 (+ 68072us) spinlock_profiling.cc:243] Waited 9.18 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a6ad21 00007fb6d9a481e3 00007fb6d99f9869 00007fb6d2ffbc44 00007fb6d2ffbbd2 00007fb6d2ffbb01
0924 00:21:27.459764 (+ 34189us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.459887 (+ 123us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.595471 (+135584us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.817590 (+222119us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.818062 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21122816}
I20260924 00:21:27.820269 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260924 00:21:27.795687 2056 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 36) took 1095 ms. Trace:
I20260924 00:21:27.821249 2056 rpcz_store.cc:276] 0924 00:21:26.700276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.394535 (+694259us) service_pool.cc:227] Handling call
0924 00:21:27.499167 (+104632us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.499364 (+ 197us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.605511 (+106147us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.795217 (+189706us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.795632 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1081088}
I20260924 00:21:27.822276 2054 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 38) took 1120 ms. Trace:
I20260924 00:21:27.822630 2054 rpcz_store.cc:276] 0924 00:21:26.701641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.430055 (+728414us) service_pool.cc:227] Handling call
0924 00:21:27.441589 (+ 11534us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.456853 (+ 15264us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.604353 (+147500us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.821768 (+217415us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.822222 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1891968}
I20260924 00:21:27.823169 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 95ac9e39e8934ecdb84ac79399dcbd9c (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260924 00:21:27.824407 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ac9e39e8934ecdb84ac79399dcbd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.824556 2054 tablet_service.cc:1467] Processing CreateTablet for tablet a4917c5d11b34f7aaa5cbdcf98709f22 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260924 00:21:27.797456 2188 tablet_bootstrap.cc:654] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.845614 1986 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 49) took 1045 ms. Trace:
I20260924 00:21:27.846035 1986 rpcz_store.cc:276] 0924 00:21:26.800576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.483849 (+683273us) service_pool.cc:227] Handling call
0924 00:21:27.549746 (+ 65897us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.549881 (+ 135us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.792175 (+242294us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.845109 (+ 52934us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.845558 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.789847 1979 tablet_service.cc:1467] Processing CreateTablet for tablet d2ef9fa9e5dd48218a5c243de0c0e49a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260924 00:21:27.799415 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 92bd85e8dadb4876827acd9ff4be377f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260924 00:21:27.801644 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 48) took 1019 ms. Trace:
I20260924 00:21:27.848289 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd85e8dadb4876827acd9ff4be377f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.803194 2058 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 37) took 1102 ms. Trace:
I20260924 00:21:27.803452 1984 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55036 (request call id 45) took 1089 ms. Trace:
I20260924 00:21:27.848791 1984 rpcz_store.cc:276] 0924 00:21:26.713649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.352877 (+639228us) service_pool.cc:227] Handling call
0924 00:21:27.385569 (+ 32692us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.385673 (+ 104us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.455135 (+ 69462us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.803109 (+347974us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.803407 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.806331 2186 raft_consensus.cc:348] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.850606 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ef9fa9e5dd48218a5c243de0c0e49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.850826 2186 raft_consensus.cc:374] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.851123 2186 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.851823 2186 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.854060 2186 ts_tablet_manager.cc:1428] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.116s user 0.004s sys 0.002s
I20260924 00:21:27.826000 2061 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 39) took 1123 ms. Trace:
I20260924 00:21:27.854592 2061 rpcz_store.cc:276] 0924 00:21:26.702257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.459070 (+756813us) service_pool.cc:227] Handling call
0924 00:21:27.501570 (+ 42500us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.501690 (+ 120us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.769006 (+267316us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.825546 (+ 56540us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.825945 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.856484 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 25490e5b518e4abf90527c9e0dd7723e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260924 00:21:27.826472 2055 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 40) took 1123 ms. Trace:
I20260924 00:21:27.848702 2058 rpcz_store.cc:276] 0924 00:21:26.700906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.395887 (+694981us) service_pool.cc:227] Handling call
0924 00:21:27.493623 (+ 97736us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.493766 (+ 143us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.595377 (+101611us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.802735 (+207358us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.803140 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5888}
I20260924 00:21:27.857568 2055 rpcz_store.cc:276] 0924 00:21:26.702977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.617026 (+914049us) service_pool.cc:227] Handling call
0924 00:21:27.629390 (+ 12364us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.629504 (+ 114us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.770249 (+140745us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.826033 (+ 55784us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.826422 (+ 389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.859138 2058 tablet_service.cc:1467] Processing CreateTablet for tablet aaea92f011f94575ab225f58d2a0db06 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260924 00:21:27.860083 2055 tablet_service.cc:1467] Processing CreateTablet for tablet e4a9343576d1478caf59e9f5629ce211 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260924 00:21:27.863122 2056 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 46) took 1111 ms. Trace:
I20260924 00:21:27.863683 2056 rpcz_store.cc:276] 0924 00:21:26.752214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.822273 (+1070059us) service_pool.cc:227] Handling call
0924 00:21:27.823957 (+ 1684us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.824048 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.839926 (+ 15878us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.862620 (+ 22694us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.863062 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.865653 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa219f0283c4632b6601e60980ae25a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260924 00:21:27.829905 2059 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 43) took 1124 ms. Trace:
I20260924 00:21:27.817404 1980 rpcz_store.cc:276] 0924 00:21:26.763249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.359489 (+596240us) service_pool.cc:227] Handling call
0924 00:21:27.415010 (+ 55521us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.452569 (+ 37559us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.583232 (+130663us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.788231 (+204999us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.788695 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.868901 2059 rpcz_store.cc:276] 0924 00:21:26.705820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.631793 (+925973us) service_pool.cc:227] Handling call
0924 00:21:27.663609 (+ 31816us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.663854 (+ 245us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.805064 (+141210us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.829414 (+ 24350us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.829849 (+ 435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.869593 2186 tablet_bootstrap.cc:492] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:27.871435 2196 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:27.826315 2126 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57740 (request call id 51) took 1014 ms. Trace:
I20260924 00:21:27.871726 2059 tablet_service.cc:1467] Processing CreateTablet for tablet c616f076fa3a4e75b7d3c51e60553d74 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260924 00:21:27.871937 2196 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:27.837484 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.874080 2196 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:27.761173 2052 tablet_service.cc:1467] Processing CreateTablet for tablet d581036c5b6e448cbae056fa4fc15f78 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260924 00:21:27.844045 2190 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:27.872524 2126 rpcz_store.cc:276] 0924 00:21:26.811655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.429378 (+617723us) service_pool.cc:227] Handling call
0924 00:21:27.431268 (+ 1890us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.431368 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.711064 (+279696us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.825852 (+114788us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.826262 (+ 410us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.875370 2190 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.875900 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:27.857754 2053 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 42) took 1153 ms. Trace:
I20260924 00:21:27.847752 1977 rpcz_store.cc:276] 0924 00:21:26.782482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.477682 (+695200us) service_pool.cc:227] Handling call
0924 00:21:27.480346 (+ 2664us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.480503 (+ 157us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.632210 (+151707us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.801091 (+168881us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.801591 (+ 500us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.801375 2185 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.884377 2053 rpcz_store.cc:276] 0924 00:21:26.704788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.620319 (+915531us) service_pool.cc:227] Handling call
0924 00:21:27.657085 (+ 36766us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.657234 (+ 149us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.805813 (+148579us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.857027 (+ 51214us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.857679 (+ 652us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.867604 2132 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57740 (request call id 53) took 1010 ms. Trace:
I20260924 00:21:27.878444 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:27.889925 1960 leader_election.cc:341] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.836784 2129 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57740 (request call id 46) took 1076 ms. Trace:
I20260924 00:21:27.889765 2053 tablet_service.cc:1467] Processing CreateTablet for tablet fc79ee9a960d47c2b7d3150a8aa37983 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260924 00:21:27.844226 2057 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 41) took 1140 ms. Trace:
I20260924 00:21:27.878014 2190 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:27.877993 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:27.880769 2202 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:27.880991 2188 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:27.893443 2188 ts_tablet_manager.cc:1397] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.020s
I20260924 00:21:27.881522 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:27.895140 2145 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:27.874082 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c616f076fa3a4e75b7d3c51e60553d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.896325 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d581036c5b6e448cbae056fa4fc15f78. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 00:21:27.882126 2110 leader_election.cc:341] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.896982 2188 raft_consensus.cc:348] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.897804 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a9343576d1478caf59e9f5629ce211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.897928 2188 raft_consensus.cc:374] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:27.898420 2188 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:27.898520 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:27.892002 2202 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:27.899513 2188 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.901414 2202 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:27.902143 2071 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:27.905465 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa219f0283c4632b6601e60980ae25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.908864 2109 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:27.910297 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:27.910688 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:27.911334 2145 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:27.911664 2190 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:27.912196 2190 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:27.912621 2190 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:27.915246 2186 tablet_bootstrap.cc:654] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.918017 2035 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:27.918983 2202 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:27.919333 2202 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:27.919661 2202 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:27.921355 2196 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:27.921862 2196 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:27.922269 2196 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:27.924296 2036 leader_election.cc:341] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.890580 2129 rpcz_store.cc:276] 0924 00:21:26.760005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.364185 (+604180us) service_pool.cc:227] Handling call
0924 00:21:27.399711 (+ 35526us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.399934 (+ 223us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.774717 (+374783us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.836285 (+ 61568us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.836722 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.891067 2057 rpcz_store.cc:276] 0924 00:21:26.703701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.551045 (+847344us) service_pool.cc:227] Handling call
0924 00:21:27.573815 (+ 22770us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.574038 (+ 223us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.799534 (+225496us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.843667 (+ 44133us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.844164 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.933269 2057 tablet_service.cc:1467] Processing CreateTablet for tablet e7be2c535b984fd6bf122044c990f72e (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260924 00:21:27.935216 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaea92f011f94575ab225f58d2a0db06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.931017 2196 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:27.937881 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:21:27.939308 2110 leader_election.cc:341] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.940073 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc79ee9a960d47c2b7d3150a8aa37983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.945472 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7be2c535b984fd6bf122044c990f72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.947484 2059 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 52) took 1107 ms. Trace:
I20260924 00:21:27.948220 2190 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:27.949041 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25490e5b518e4abf90527c9e0dd7723e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.951629 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4917c5d11b34f7aaa5cbdcf98709f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.948033 2059 rpcz_store.cc:276] 0924 00:21:26.840008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.870058 (+1030050us) service_pool.cc:227] Handling call
0924 00:21:27.872599 (+ 2541us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.872690 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.936667 (+ 63977us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.946709 (+ 10042us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.947397 (+ 688us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.888752 2132 rpcz_store.cc:276] 0924 00:21:26.857176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.552362 (+695186us) service_pool.cc:227] Handling call
0924 00:21:27.584392 (+ 32030us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.584572 (+ 180us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.815028 (+230456us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.866932 (+ 51904us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.867526 (+ 594us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.961360 2185 ts_tablet_manager.cc:1428] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.230s user 0.015s sys 0.006s
I20260924 00:21:27.962530 2185 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
W20260924 00:21:27.966542 2097 tablet.cc:2369] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:21:27.972599 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:27.975474 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:27.976131 2185 tablet_bootstrap.cc:654] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:27.973335 2196 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:21:27.978622 1960 leader_election.cc:341] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:27.986752 2053 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 55) took 1101 ms. Trace:
I20260924 00:21:27.987193 2053 rpcz_store.cc:276] 0924 00:21:26.885454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.888039 (+1002585us) service_pool.cc:227] Handling call
0924 00:21:27.890931 (+ 2892us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.891121 (+ 190us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.954474 (+ 63353us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.986226 (+ 31752us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.986689 (+ 463us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.990478 2059 tablet_service.cc:1467] Processing CreateTablet for tablet d1caddf844ef49c0b3664f36bb76b869 (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260924 00:21:27.992409 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1caddf844ef49c0b3664f36bb76b869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:27.998062 2055 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 50) took 1194 ms. Trace:
I20260924 00:21:27.998435 2055 rpcz_store.cc:276] 0924 00:21:26.803321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.858844 (+1055523us) service_pool.cc:227] Handling call
0924 00:21:27.861226 (+ 2382us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.861504 (+ 278us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.919804 (+ 58300us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:27.996119 (+ 76315us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:27.998006 (+ 1887us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:27.999351 2186 tablet_bootstrap.cc:492] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:27.999995 2186 ts_tablet_manager.cc:1397] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.131s user 0.012s sys 0.000s
I20260924 00:21:27.997910 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:28.001387 2145 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.001317 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 832ee611b00d4a42b3245beb345af7ad (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260924 00:21:28.002910 2053 tablet_service.cc:1467] Processing CreateTablet for tablet d2ef9fa9e5dd48218a5c243de0c0e49a (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260924 00:21:28.004092 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832ee611b00d4a42b3245beb345af7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:28.003547 2186 raft_consensus.cc:348] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.004536 2186 raft_consensus.cc:374] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.000012 2190 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.006940 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ef9fa9e5dd48218a5c243de0c0e49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:28.008608 2145 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:28.009326 2056 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 51) took 1197 ms. Trace:
I20260924 00:21:28.004930 2186 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.009688 2056 rpcz_store.cc:276] 0924 00:21:26.812032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.864714 (+1052682us) service_pool.cc:227] Handling call
0924 00:21:27.873214 (+ 8500us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.873303 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.970475 (+ 97172us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.005778 (+ 35303us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.009269 (+ 3491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.010852 2186 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.025084 2186 ts_tablet_manager.cc:1428] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20260924 00:21:28.026705 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.030644 2190 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.030812 2186 tablet_bootstrap.cc:492] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.031162 2190 raft_consensus.cc:686] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:28.032002 2190 consensus_queue.cc:237] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.012348 2202 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:28.033897 2071 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:28.027189 2060 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 45) took 1286 ms. Trace:
I20260924 00:21:28.034994 2060 rpcz_store.cc:276] 0924 00:21:26.740951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.819354 (+1078403us) service_pool.cc:227] Handling call
0924 00:21:27.821046 (+ 1692us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.832068 (+ 11022us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.942901 (+110833us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.026678 (+ 83777us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.027131 (+ 453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.035643 2109 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.036450 2196 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.037895 2058 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 49) took 1238 ms. Trace:
I20260924 00:21:28.038287 2058 rpcz_store.cc:276] 0924 00:21:26.799857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.858190 (+1058333us) service_pool.cc:227] Handling call
0924 00:21:27.860024 (+ 1834us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.860131 (+ 107us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.989304 (+129173us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.029052 (+ 39748us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.037836 (+ 8784us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.039327 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:28.040973 2202 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:28.041559 2202 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.041983 2202 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.043367 2185 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.044025 2185 ts_tablet_manager.cc:1397] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.000s
W20260924 00:21:28.044919 2036 leader_election.cc:341] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.045833 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.046468 1996 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:28.047662 2036 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:28.048496 2061 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 48) took 1250 ms. Trace:
I20260924 00:21:28.048856 2061 rpcz_store.cc:276] 0924 00:21:26.797709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.855583 (+1057874us) service_pool.cc:227] Handling call
0924 00:21:27.860510 (+ 4927us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.860596 (+ 86us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.981420 (+120824us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.048016 (+ 66596us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.048428 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.050063 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:28.051600 2035 leader_election.cc:341] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.051463 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 92bd85e8dadb4876827acd9ff4be377f (DEFAULT_TABLE table=test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260924 00:21:28.053462 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd85e8dadb4876827acd9ff4be377f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:28.054751 2188 ts_tablet_manager.cc:1428] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.161s user 0.067s sys 0.077s
I20260924 00:21:28.054999 2057 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 53) took 1185 ms. Trace:
I20260924 00:21:28.055382 2057 rpcz_store.cc:276] 0924 00:21:26.869891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.932346 (+1062455us) service_pool.cc:227] Handling call
0924 00:21:27.934078 (+ 1732us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.934172 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.964932 (+ 30760us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.054334 (+ 89402us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.054840 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.056242 2188 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.058547 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:28.057966 2052 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 44) took 1343 ms. Trace:
I20260924 00:21:28.059633 2052 rpcz_store.cc:276] 0924 00:21:26.715166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.760242 (+1045076us) service_pool.cc:227] Handling call
0924 00:21:27.874905 (+114663us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.875000 (+ 95us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.970979 (+ 95979us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.057479 (+ 86500us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.057910 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.052523 2054 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 47) took 1281 ms. Trace:
I20260924 00:21:28.052870 2202 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.061544 2054 rpcz_store.cc:276] 0924 00:21:26.766787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.823611 (+1056824us) service_pool.cc:227] Handling call
0924 00:21:27.853550 (+ 29939us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.853720 (+ 170us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:27.979353 (+125633us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.047801 (+ 68448us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.048428 (+ 627us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.063261 2145 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:28.064776 2035 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:28.062243 2202 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.065528 2202 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.065912 2202 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.064779 2185 raft_consensus.cc:348] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.071492 2185 raft_consensus.cc:374] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.071897 2185 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.072715 2185 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.075157 2185 ts_tablet_manager.cc:1428] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.031s user 0.009s sys 0.000s
I20260924 00:21:28.076529 2186 tablet_bootstrap.cc:654] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.079214 2212 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.080194 2056 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 58) took 1096 ms. Trace:
I20260924 00:21:28.080722 2056 rpcz_store.cc:276] 0924 00:21:26.984345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:28.021226 (+1036881us) service_pool.cc:227] Handling call
0924 00:21:28.052241 (+ 31015us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:28.052348 (+ 107us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:28.068666 (+ 16318us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.076390 (+ 7724us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.080116 (+ 3726us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.080505 2185 tablet_bootstrap.cc:492] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.087561 2188 tablet_bootstrap.cc:654] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.091575 1894 catalog_manager.cc:5495] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:28.091063 2185 tablet_bootstrap.cc:654] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.072664 2202 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.100245 2059 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 54) took 1230 ms. Trace:
I20260924 00:21:28.101066 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:28.100811 2059 rpcz_store.cc:276] 0924 00:21:26.870309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.989085 (+1118776us) service_pool.cc:227] Handling call
0924 00:21:27.991755 (+ 2670us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:27.991898 (+ 143us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:28.065612 (+ 73714us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.099041 (+ 33429us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.100163 (+ 1122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.102172 1996 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.105417 2202 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.106204 2055 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 57) took 1150 ms. Trace:
I20260924 00:21:28.106477 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:28.107316 2053 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:47762 (request call id 56) took 1197 ms. Trace:
I20260924 00:21:28.106674 2055 rpcz_store.cc:276] 0924 00:21:26.956123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:28.000175 (+1044052us) service_pool.cc:227] Handling call
0924 00:21:28.002219 (+ 2044us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:28.003697 (+ 1478us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:28.089842 (+ 86145us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.105516 (+ 15674us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.106148 (+ 632us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260924 00:21:28.107913 2035 leader_election.cc:341] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.107841 2053 rpcz_store.cc:276] 0924 00:21:26.909505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:27.992219 (+1082714us) service_pool.cc:227] Handling call
0924 00:21:28.006388 (+ 14169us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0924 00:21:28.006486 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0924 00:21:28.076444 (+ 69958us) tablet_metadata.cc:641] Metadata flushed
0924 00:21:28.106664 (+ 30220us) tablet_replica.cc:205] Creating consensus instance
0924 00:21:28.107245 (+ 581us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:21:28.113168 1996 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:28.114192 2186 tablet_bootstrap.cc:492] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.114321 2036 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:28.115140 2186 ts_tablet_manager.cc:1397] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.008s
I20260924 00:21:28.116318 2202 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.119432 2188 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.119975 2188 ts_tablet_manager.cc:1397] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.064s user 0.028s sys 0.009s
I20260924 00:21:28.119925 2202 raft_consensus.cc:686] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:28.120990 2202 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.123176 2188 raft_consensus.cc:348] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.124015 2188 raft_consensus.cc:374] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.124411 2188 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.124802 2185 tablet_bootstrap.cc:492] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.125460 2185 ts_tablet_manager.cc:1397] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.009s
I20260924 00:21:28.125357 2188 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.128136 2188 ts_tablet_manager.cc:1428] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260924 00:21:28.128237 2186 raft_consensus.cc:348] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.128345 2185 raft_consensus.cc:348] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.129061 2186 raft_consensus.cc:374] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.129274 2185 raft_consensus.cc:374] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.129451 2188 tablet_bootstrap.cc:492] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.129709 2185 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.130570 2185 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.132649 2185 ts_tablet_manager.cc:1428] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:21:28.133641 2185 tablet_bootstrap.cc:492] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.129599 2186 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.134752 2186 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.136572 2188 tablet_bootstrap.cc:654] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.136085 1893 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:28.139662 2196 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.140408 2196 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.142532 2196 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.145236 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.145480 2185 tablet_bootstrap.cc:654] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.145474 2186 ts_tablet_manager.cc:1428] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260924 00:21:28.145987 2071 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:28.146895 2186 tablet_bootstrap.cc:492] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.147291 2109 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:28.148315 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:21:28.149745 2110 leader_election.cc:341] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.152328 2196 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.152715 2196 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.153743 2186 tablet_bootstrap.cc:654] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.153043 2196 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.161691 2188 tablet_bootstrap.cc:492] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.163878 2188 ts_tablet_manager.cc:1397] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.024s
I20260924 00:21:28.166810 2188 raft_consensus.cc:348] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.167084 2196 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.167896 2188 raft_consensus.cc:374] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.168279 2188 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.169064 2188 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.170375 2186 tablet_bootstrap.cc:492] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.170228 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:28.171036 2186 ts_tablet_manager.cc:1397] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
W20260924 00:21:28.171973 2110 leader_election.cc:341] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.172462 2188 ts_tablet_manager.cc:1428] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:21:28.173708 2188 tablet_bootstrap.cc:492] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.174719 2186 raft_consensus.cc:348] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.175762 2186 raft_consensus.cc:374] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.176151 2186 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.178335 2196 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.180773 2196 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.178241 2186 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.180788 2185 tablet_bootstrap.cc:492] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.181360 2196 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.182266 2185 ts_tablet_manager.cc:1397] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.017s
I20260924 00:21:28.183594 2186 ts_tablet_manager.cc:1428] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
I20260924 00:21:28.184072 2196 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.184836 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.185532 1996 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:28.186749 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:28.188426 2185 raft_consensus.cc:348] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.189532 2185 raft_consensus.cc:374] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.191062 2185 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.192595 2188 tablet_bootstrap.cc:654] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.194211 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:28.194852 2071 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.195982 2186 tablet_bootstrap.cc:492] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.187695 2227 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.191773 2185 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.198956 2227 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.199331 2227 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.200611 2185 ts_tablet_manager.cc:1428] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260924 00:21:28.201663 2185 tablet_bootstrap.cc:492] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.202044 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.204061 2071 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:28.208067 2185 tablet_bootstrap.cc:654] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:28.209517 2109 leader_election.cc:341] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.210539 2109 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:28.211401 2190 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.211931 2190 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.212913 2211 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.213413 2211 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.215303 2196 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.215723 2211 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.217149 2190 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.217545 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.219122 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.219754 2070 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:28.221604 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:21:28.222525 1960 leader_election.cc:341] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.223966 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.224623 2145 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:28.224783 2227 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.225795 2190 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.226194 2190 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.226529 2190 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.227504 2227 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.228503 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:28.228721 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:28.229094 1996 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.233361 2211 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.233855 2211 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.234181 2211 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.237658 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.242311 2196 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:28.243021 1996 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:28.243229 2196 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.244179 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:21:28.245122 1960 leader_election.cc:341] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.247673 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:21:28.249936 2109 leader_election.cc:341] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.250936 2231 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.251446 2231 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:28.252627 2231 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.254916 2190 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.262796 2186 tablet_bootstrap.cc:654] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.263034 1894 catalog_manager.cc:5495] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:28.286736 2211 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.290074 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:28.291592 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:28.291889 2185 tablet_bootstrap.cc:492] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
W20260924 00:21:28.291958 1960 leader_election.cc:341] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.292181 2071 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.292629 2185 ts_tablet_manager.cc:1397] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.091s user 0.041s sys 0.036s
I20260924 00:21:28.294688 2185 raft_consensus.cc:348] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:28.295123 2185 raft_consensus.cc:374] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.295271 2185 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.295373 2190 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.296069 2211 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.296466 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:28.298316 2185 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:28.300297 2185 ts_tablet_manager.cc:1428] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260924 00:21:28.302671 2188 tablet_bootstrap.cc:492] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.304164 2185 tablet_bootstrap.cc:492] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
W20260924 00:21:28.303229 1960 leader_election.cc:341] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.303480 2071 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:28.305296 2188 ts_tablet_manager.cc:1397] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.131s user 0.034s sys 0.042s
I20260924 00:21:28.307252 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:28.305249 1894 catalog_manager.cc:5495] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:28.307842 2145 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.309468 2185 tablet_bootstrap.cc:654] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.310957 2188 raft_consensus.cc:348] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.312227 2188 raft_consensus.cc:374] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.312592 2188 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.313411 2188 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.311931 2186 tablet_bootstrap.cc:492] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.314970 2186 ts_tablet_manager.cc:1397] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.119s user 0.018s sys 0.003s
I20260924 00:21:28.315202 2145 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:28.305881 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:28.316530 2211 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.317008 2211 raft_consensus.cc:686] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:28.318065 2211 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.322315 2185 tablet_bootstrap.cc:492] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.318585 2186 raft_consensus.cc:348] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.322790 2185 ts_tablet_manager.cc:1397] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20260924 00:21:28.323158 2186 raft_consensus.cc:374] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.323609 2186 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.324633 2186 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.325563 2185 raft_consensus.cc:348] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.326464 2185 raft_consensus.cc:374] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.326545 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.326817 2185 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.327348 2190 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.327577 2185 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.331102 2186 ts_tablet_manager.cc:1428] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260924 00:21:28.332127 2186 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.323148 2188 ts_tablet_manager.cc:1428] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260924 00:21:28.334054 2188 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.338578 2190 raft_consensus.cc:686] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:28.340132 2188 tablet_bootstrap.cc:654] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.339810 2190 consensus_queue.cc:237] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.345753 2186 tablet_bootstrap.cc:654] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.357534 1893 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:28.386165 1893 catalog_manager.cc:5495] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:28.390074 2231 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.390762 2231 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.393347 2185 ts_tablet_manager.cc:1428] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.070s user 0.026s sys 0.036s
I20260924 00:21:28.394316 2185 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.394882 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.396018 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:21:28.396420 2109 leader_election.cc:341] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:28.398010 2110 leader_election.cc:341] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.398599 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.399693 2196 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.400521 2231 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.402788 2185 tablet_bootstrap.cc:654] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.409183 2188 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.409682 2186 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.409804 2188 ts_tablet_manager.cc:1397] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.076s user 0.038s sys 0.016s
I20260924 00:21:28.410375 2186 ts_tablet_manager.cc:1397] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.078s user 0.035s sys 0.031s
I20260924 00:21:28.413028 2186 raft_consensus.cc:348] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.413123 2188 raft_consensus.cc:348] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.413800 2186 raft_consensus.cc:374] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.413872 2188 raft_consensus.cc:374] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.414166 2186 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.414222 2188 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.414415 2185 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.415033 2185 ts_tablet_manager.cc:1397] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20260924 00:21:28.414968 2186 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.415171 2188 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.416882 2186 ts_tablet_manager.cc:1428] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 00:21:28.417474 2188 ts_tablet_manager.cc:1428] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 00:21:28.418391 2188 tablet_bootstrap.cc:492] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.419068 2186 tablet_bootstrap.cc:492] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.421077 2185 raft_consensus.cc:348] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.421876 2185 raft_consensus.cc:374] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.422147 2185 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.422863 2185 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.424911 2188 tablet_bootstrap.cc:654] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.424912 2186 tablet_bootstrap.cc:654] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.424949 2185 ts_tablet_manager.cc:1428] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260924 00:21:28.426321 2185 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.433487 2185 tablet_bootstrap.cc:654] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.441284 2238 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:28.451503 2185 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.452869 2185 ts_tablet_manager.cc:1397] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.000s
I20260924 00:21:28.456140 2185 raft_consensus.cc:348] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.460819 2185 raft_consensus.cc:374] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.461292 2185 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.462798 2185 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.467514 2185 ts_tablet_manager.cc:1428] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260924 00:21:28.474853 2185 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.483780 2071 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.488103 2185 tablet_bootstrap.cc:654] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.497469 2238 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:28.500808 2185 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.501307 2185 ts_tablet_manager.cc:1397] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.002s
I20260924 00:21:28.504527 2185 raft_consensus.cc:348] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.505610 2185 raft_consensus.cc:374] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.505957 2185 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.506883 2185 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.510664 2185 ts_tablet_manager.cc:1428] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 00:21:28.511225 2186 tablet_bootstrap.cc:492] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.511716 2186 ts_tablet_manager.cc:1397] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.093s user 0.035s sys 0.022s
I20260924 00:21:28.513180 2188 tablet_bootstrap.cc:492] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.513679 2188 ts_tablet_manager.cc:1397] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.095s user 0.042s sys 0.039s
I20260924 00:21:28.514117 2186 raft_consensus.cc:348] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.513108 2185 tablet_bootstrap.cc:492] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.514839 2186 raft_consensus.cc:374] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.515296 2186 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.516067 2186 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.516503 2188 raft_consensus.cc:348] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.517231 2188 raft_consensus.cc:374] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.517501 2188 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.518190 2188 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.520632 2185 tablet_bootstrap.cc:654] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.520788 2188 ts_tablet_manager.cc:1428] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:21:28.521797 2188 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.526873 2188 tablet_bootstrap.cc:654] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.528348 2185 tablet_bootstrap.cc:492] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.528927 2185 ts_tablet_manager.cc:1397] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260924 00:21:28.531826 2185 raft_consensus.cc:348] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.532840 2185 raft_consensus.cc:374] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.533285 2185 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
W20260924 00:21:28.534585 2035 consensus_peers.cc:487] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Couldn't send request to peer 8c129c01e1c047628f6a87b001aa7258. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:28.534169 2185 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.536336 2186 ts_tablet_manager.cc:1428] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20260924 00:21:28.537047 2185 ts_tablet_manager.cc:1428] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:21:28.538424 2186 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.540710 2185 tablet_bootstrap.cc:492] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.545050 2186 tablet_bootstrap.cc:654] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.548815 2185 tablet_bootstrap.cc:654] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.554265 2188 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.554711 2188 ts_tablet_manager.cc:1397] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.009s
I20260924 00:21:28.557464 2186 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.557360 2188 raft_consensus.cc:348] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.558087 2188 raft_consensus.cc:374] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.558264 2186 ts_tablet_manager.cc:1397] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s
I20260924 00:21:28.558367 2188 raft_consensus.cc:729] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.559883 2188 consensus_queue.cc:260] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.561583 2185 tablet_bootstrap.cc:492] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.561262 2186 raft_consensus.cc:348] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.562004 2185 ts_tablet_manager.cc:1397] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20260924 00:21:28.562053 2186 raft_consensus.cc:374] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.562762 2186 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.563446 2188 ts_tablet_manager.cc:1428] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260924 00:21:28.563383 2186 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.564350 2188 tablet_bootstrap.cc:492] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.566398 2186 ts_tablet_manager.cc:1428] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260924 00:21:28.569036 2186 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.568847 2185 raft_consensus.cc:348] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.569783 2185 raft_consensus.cc:374] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.570112 2185 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.571414 2188 tablet_bootstrap.cc:654] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.570878 2185 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.579281 2211 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.579914 2211 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.582397 2211 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.583163 2186 tablet_bootstrap.cc:654] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.585309 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.586956 2231 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.587420 2231 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:28.587911 1960 leader_election.cc:341] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.589411 2185 ts_tablet_manager.cc:1428] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260924 00:21:28.589531 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.590907 2185 tablet_bootstrap.cc:492] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
W20260924 00:21:28.592089 1960 leader_election.cc:341] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.592577 1960 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.594216 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.595117 1996 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:28.595927 2211 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.598003 2202 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.598230 2231 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.598687 2202 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.600244 2185 tablet_bootstrap.cc:654] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.600668 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.606484 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.605207 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.606927 2110 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:28.602842 2202 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:28.607898 2109 leader_election.cc:341] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.608314 1996 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
W20260924 00:21:28.608848 2035 leader_election.cc:341] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.609066 2231 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.609444 2186 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.609441 2231 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.609876 2211 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.609897 2231 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.609930 2186 ts_tablet_manager.cc:1397] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.003s
I20260924 00:21:28.610415 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:28.610621 2211 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.614814 2202 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.615156 2202 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.615475 2202 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.615929 2211 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.614212 2188 tablet_bootstrap.cc:492] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.616622 2188 ts_tablet_manager.cc:1397] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.052s user 0.020s sys 0.024s
I20260924 00:21:28.617300 2186 raft_consensus.cc:348] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.617975 2186 raft_consensus.cc:374] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.618263 2186 raft_consensus.cc:729] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.624600 2231 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.626536 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.624688 2186 consensus_queue.cc:260] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.625797 2188 raft_consensus.cc:348] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.626356 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.628158 2188 raft_consensus.cc:374] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.628481 2188 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.628616 2071 raft_consensus.cc:2379] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:28.630244 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:28.630582 2186 ts_tablet_manager.cc:1428] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.016s user 0.006s sys 0.003s
I20260924 00:21:28.627152 2202 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.630885 1996 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.671648 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:28.631064 2188 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.674691 2196 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:28.676016 2185 tablet_bootstrap.cc:492] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.676478 2185 ts_tablet_manager.cc:1397] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.012s
I20260924 00:21:28.642781 2231 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.631343 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.679358 2185 raft_consensus.cc:348] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.680274 2185 raft_consensus.cc:374] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.680526 2185 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.680594 2202 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.681663 2186 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.681324 2185 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:28.682364 2109 leader_election.cc:341] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.682910 2188 ts_tablet_manager.cc:1428] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.066s user 0.020s sys 0.020s
I20260924 00:21:28.683847 2188 tablet_bootstrap.cc:492] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.684886 2211 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.686542 2185 ts_tablet_manager.cc:1428] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260924 00:21:28.687547 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:21:28.687161 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 55953.4us
I20260924 00:21:28.689944 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:21:28.690754 2035 leader_election.cc:341] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.690937 1995 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.691082 2185 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.696339 2211 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.701018 2186 tablet_bootstrap.cc:654] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.700865 1995 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:28.703275 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:28.704281 2261 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260924 00:21:28.704789 2259 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.705452 2259 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:28.705782 2188 tablet_bootstrap.cc:654] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.707602 2185 tablet_bootstrap.cc:654] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.707451 2259 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:28.711170 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:21:28.712707 2202 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.713332 2070 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.713407 2202 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.716667 1996 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:28.717851 2110 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.718003 2202 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.718966 2231 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.719437 2231 raft_consensus.cc:686] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:28.722947 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.724409 2231 consensus_queue.cc:237] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.728204 2211 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:28.729552 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.730327 2145 raft_consensus.cc:2379] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:28.731379 2238 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.731808 2238 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.738032 2036 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:28.734282 2238 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.739300 2202 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.749373 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.757275 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.759074 2186 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:28.759567 2186 ts_tablet_manager.cc:1397] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.078s user 0.014s sys 0.004s
I20260924 00:21:28.753770 2231 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260924 00:21:28.775249 2185 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.775857 2185 ts_tablet_manager.cc:1397] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.085s user 0.020s sys 0.037s
I20260924 00:21:28.780155 2185 raft_consensus.cc:348] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.781577 2185 raft_consensus.cc:374] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.781934 2185 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.782912 2185 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:28.770485 1960 leader_election.cc:341] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.775799 2268 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.792557 2186 raft_consensus.cc:348] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.796829 2186 raft_consensus.cc:374] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.797307 2186 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.802172 2185 ts_tablet_manager.cc:1428] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.026s user 0.004s sys 0.004s
I20260924 00:21:28.802922 2185 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.801916 2186 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.799017 2238 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.799377 2259 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:21:28.799252 1960 leader_election.cc:341] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.806926 2186 ts_tablet_manager.cc:1428] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.047s user 0.010s sys 0.000s
I20260924 00:21:28.795295 2268 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.808092 2259 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.812891 2186 tablet_bootstrap.cc:492] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:28.815132 2202 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.809212 1960 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.816812 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.817492 1995 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:28.809288 2238 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:28.819114 2259 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.819831 2238 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.820669 2110 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:28.822914 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.823599 2145 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:28.825289 2035 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:28.826426 2202 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.827018 2202 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.827575 2202 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.822274 2238 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.825344 1894 catalog_manager.cc:5495] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:28.836750 2269 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.836802 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.837288 2269 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.837652 2269 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.837808 2070 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:28.838017 2188 tablet_bootstrap.cc:492] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:28.838439 2188 ts_tablet_manager.cc:1397] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.155s user 0.009s sys 0.028s
I20260924 00:21:28.840081 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.840837 1995 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:28.843600 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.844717 2185 tablet_bootstrap.cc:654] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:28.845515 2109 leader_election.cc:341] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.850875 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.851681 2145 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:28.853292 2268 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.854493 2188 raft_consensus.cc:348] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.859753 2188 raft_consensus.cc:374] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 00:21:28.857954 1960 consensus_peers.cc:487] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd -> Peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Couldn't send request to peer 8c129c01e1c047628f6a87b001aa7258. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:28.860854 2188 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.862527 2188 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.869925 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.871217 2238 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.871783 2202 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.876133 2186 tablet_bootstrap.cc:654] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:28.879767 2211 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:28.881567 2202 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.876876 2231 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260924 00:21:28.878609 2274 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260924 00:21:28.885972 2188 ts_tablet_manager.cc:1428] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.047s user 0.010s sys 0.000s
I20260924 00:21:28.895088 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:28.895758 1996 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.903525 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:28.906978 2144 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.916517 2259 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.917240 2259 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.921839 2259 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.938082 2188 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:28.937557 1894 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:28.942150 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.943033 1993 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:28.932087 2238 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.945992 1996 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:28.948863 2036 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:28.950093 2185 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:28.950398 2202 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:28.950608 2185 ts_tablet_manager.cc:1397] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.148s user 0.015s sys 0.020s
I20260924 00:21:28.951260 2202 raft_consensus.cc:686] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:28.952071 2238 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.955266 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.958212 2142 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.961184 2035 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:28.961839 2144 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:28.962597 2259 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.955986 2238 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.965446 2238 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.960649 2202 consensus_queue.cc:237] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.966862 2261 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.968667 2185 raft_consensus.cc:348] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.973184 2261 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.954209 2269 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.975638 2261 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:28.976017 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:28.976420 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:28.977054 1995 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.977680 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.979525 2238 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:28.980424 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:28.973659 2185 raft_consensus.cc:374] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:28.981127 2185 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:28.981189 1994 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:28.983031 2185 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.982622 2269 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.984784 2211 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:28.985661 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:28.985334 2211 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:28.986478 2185 ts_tablet_manager.cc:1428] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.035s user 0.008s sys 0.002s
I20260924 00:21:28.980499 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:28.987257 2109 leader_election.cc:341] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.987886 2211 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:28.988108 1960 leader_election.cc:341] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.988710 2185 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:28.987572 2144 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:28.989996 2035 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:21:28.990540 1960 leader_election.cc:341] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:28.990942 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:28.993312 2261 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:28.993723 2261 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:28.994066 2261 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:28.994441 2238 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.994705 2259 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:28.996232 2188 tablet_bootstrap.cc:654] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.001648 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:28.999078 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:29.012084 1960 leader_election.cc:341] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:29.012789 1960 leader_election.cc:341] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.013202 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.014137 2186 tablet_bootstrap.cc:492] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:29.014263 2273 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.014608 2186 ts_tablet_manager.cc:1397] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.202s user 0.014s sys 0.008s
I20260924 00:21:29.014917 2273 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.016744 2211 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.019268 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.021687 2273 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:28.999380 2185 tablet_bootstrap.cc:654] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.022773 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.024932 2109 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.026535 2273 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.021595 1995 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:29.028752 2110 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.029592 2273 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.030315 2273 raft_consensus.cc:686] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:29.031250 2273 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.038740 2261 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.017040 2186 raft_consensus.cc:348] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.041842 2186 raft_consensus.cc:374] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.042169 2186 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.042389 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:29.043008 1995 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.044548 2269 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:29.045065 2269 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.045719 2186 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.051256 2186 ts_tablet_manager.cc:1428] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.036s user 0.005s sys 0.003s
I20260924 00:21:29.052300 2186 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:29.052727 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.054258 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.054637 2269 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.055928 2109 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.056780 2269 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:29.058607 2185 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.058640 2186 tablet_bootstrap.cc:654] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.059098 2185 ts_tablet_manager.cc:1397] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.071s user 0.021s sys 0.004s
I20260924 00:21:29.059615 2261 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.061162 2188 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.061342 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:29.061668 2188 ts_tablet_manager.cc:1397] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.010s
I20260924 00:21:29.061952 2142 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.062965 2185 raft_consensus.cc:348] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.063666 2185 raft_consensus.cc:374] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.063915 2185 raft_consensus.cc:729] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.058360 1892 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:29.064646 2185 consensus_queue.cc:260] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.066761 2185 ts_tablet_manager.cc:1428] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 00:21:29.067665 2185 tablet_bootstrap.cc:492] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.069969 2142 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:29.071400 2035 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:29.072218 2261 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.072696 2261 raft_consensus.cc:686] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:29.073587 2261 consensus_queue.cc:237] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.075002 2185 tablet_bootstrap.cc:654] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.078768 2190 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.079457 2190 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.082054 2190 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.111132 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:29.114568 1960 leader_election.cc:341] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.116739 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:29.125226 1960 leader_election.cc:341] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.126530 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.127836 2269 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.129491 2142 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.129812 2190 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.138854 1893 catalog_manager.cc:5495] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:29.144397 2188 raft_consensus.cc:348] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.148864 2274 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.149382 2274 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.149968 2273 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.148808 2188 raft_consensus.cc:374] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.150842 2188 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.151396 2274 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.151744 2188 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.153834 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.158134 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.160426 1994 raft_consensus.cc:2379] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:29.164000 2110 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.165366 2269 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.170079 1995 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:29.172467 2188 ts_tablet_manager.cc:1428] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.110s user 0.006s sys 0.002s
I20260924 00:21:29.172676 2269 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.173473 2188 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.174515 2186 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:29.174925 2186 ts_tablet_manager.cc:1397] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.123s user 0.042s sys 0.051s
I20260924 00:21:29.176856 2190 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.177004 2185 tablet_bootstrap.cc:492] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.177834 2185 ts_tablet_manager.cc:1397] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.110s user 0.029s sys 0.040s
I20260924 00:21:29.177649 2190 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.180277 2185 raft_consensus.cc:348] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.181031 2185 raft_consensus.cc:374] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.181329 2185 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.181967 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.181962 2185 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:29.184641 1960 leader_election.cc:341] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.185778 2186 raft_consensus.cc:348] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.186646 2186 raft_consensus.cc:374] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.186929 2186 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.187677 2186 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.190047 2261 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:29.188194 2190 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.191634 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:29.194314 2186 ts_tablet_manager.cc:1428] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.019s user 0.006s sys 0.003s
I20260924 00:21:29.195391 2186 tablet_bootstrap.cc:492] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
W20260924 00:21:29.198994 1960 leader_election.cc:341] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.199961 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.201829 2186 tablet_bootstrap.cc:654] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.202132 2190 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.205924 2188 tablet_bootstrap.cc:654] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.214617 2269 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.215194 2269 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.217540 2269 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.219925 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:21:29.222540 2110 leader_election.cc:341] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.223031 2185 ts_tablet_manager.cc:1428] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260924 00:21:29.223506 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.224068 2185 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.224188 2071 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:29.225479 2109 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:29.227166 2274 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.227684 2274 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.228502 2274 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.231341 2185 tablet_bootstrap.cc:654] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.237010 2188 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.237560 2188 ts_tablet_manager.cc:1397] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.064s user 0.015s sys 0.003s
I20260924 00:21:29.237303 2274 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.240478 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:29.240420 2188 raft_consensus.cc:348] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.241369 2188 raft_consensus.cc:374] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.241643 2188 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.242347 2188 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:29.246930 2110 leader_election.cc:341] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.250125 2274 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.250244 2202 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.250816 2202 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.253753 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:29.254359 2071 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.256916 2202 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.257380 2188 ts_tablet_manager.cc:1428] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260924 00:21:29.258260 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.258972 1996 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:29.259192 2188 tablet_bootstrap.cc:492] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.258284 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:29.260845 2035 leader_election.cc:341] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.261960 2186 tablet_bootstrap.cc:492] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:29.261929 2036 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:29.262655 2186 ts_tablet_manager.cc:1397] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.067s user 0.017s sys 0.005s
I20260924 00:21:29.263347 2202 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.263796 2202 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.264194 2202 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.267937 2186 raft_consensus.cc:348] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.268692 2186 raft_consensus.cc:374] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.268498 2261 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.269016 2186 raft_consensus.cc:729] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.273025 2186 consensus_queue.cc:260] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.273828 2190 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.274318 2190 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.272061 1893 catalog_manager.cc:5495] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:29.279543 2186 ts_tablet_manager.cc:1428] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260924 00:21:29.280524 2186 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:29.286531 2190 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.287418 2186 tablet_bootstrap.cc:654] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.287334 2274 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.287973 2274 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.289356 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.290799 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.291106 2274 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.291479 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.292085 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:29.292822 2110 leader_election.cc:341] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.293318 2142 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
W20260924 00:21:29.293740 2109 leader_election.cc:341] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:29.294114 1960 leader_election.cc:341] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.294377 2109 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.295511 2274 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.298126 2188 tablet_bootstrap.cc:654] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.298674 2071 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:29.298974 2202 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.302601 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.302613 2109 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:29.304297 2274 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.305445 2274 raft_consensus.cc:686] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:29.304739 2190 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.306638 2274 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.307502 2190 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.307917 2190 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.330786 2202 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.326050 2190 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.332815 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:29.333670 1995 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.335023 2190 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.336941 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:29.337684 2144 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.338084 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:29.339623 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:21:29.341935 1960 leader_election.cc:341] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.344403 2185 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.345032 2185 ts_tablet_manager.cc:1397] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.121s user 0.014s sys 0.027s
W20260924 00:21:29.345566 2035 leader_election.cc:341] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.351783 2186 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:29.352442 2186 ts_tablet_manager.cc:1397] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.006s
I20260924 00:21:29.353796 2144 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:29.355162 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.355330 2185 raft_consensus.cc:348] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.356374 2185 raft_consensus.cc:374] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.356477 2211 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.356801 2185 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.357152 2211 raft_consensus.cc:686] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:29.357758 2185 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.358049 2211 consensus_queue.cc:237] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.360747 2185 ts_tablet_manager.cc:1428] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20260924 00:21:29.362057 2185 tablet_bootstrap.cc:492] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.364522 2260 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.366247 2186 raft_consensus.cc:348] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.367192 2186 raft_consensus.cc:374] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.367519 2186 raft_consensus.cc:729] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.374008 2188 tablet_bootstrap.cc:492] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.374498 2188 ts_tablet_manager.cc:1397] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.115s user 0.012s sys 0.017s
I20260924 00:21:29.368327 2186 consensus_queue.cc:260] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.377600 2186 ts_tablet_manager.cc:1428] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20260924 00:21:29.378679 2186 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:29.384894 2186 tablet_bootstrap.cc:654] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.386498 2188 raft_consensus.cc:348] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.391831 2188 raft_consensus.cc:374] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.392304 2188 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.393193 2188 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.396046 2188 ts_tablet_manager.cc:1428] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260924 00:21:29.373636 1995 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:29.388587 1891 catalog_manager.cc:5495] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:29.390794 2185 tablet_bootstrap.cc:654] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.397794 2188 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.402087 2036 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:29.405508 2202 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.406301 2202 raft_consensus.cc:686] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:29.407613 2188 tablet_bootstrap.cc:654] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.407316 2202 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.423569 2259 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.435187 2185 tablet_bootstrap.cc:492] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.435877 2185 ts_tablet_manager.cc:1397] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.074s user 0.016s sys 0.001s
I20260924 00:21:29.439785 2185 raft_consensus.cc:348] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.440753 2185 raft_consensus.cc:374] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.441071 2185 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.442986 1892 catalog_manager.cc:5495] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:29.444479 2188 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.445443 2188 ts_tablet_manager.cc:1397] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.005s
I20260924 00:21:29.448132 2186 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:29.448288 2188 raft_consensus.cc:348] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.448823 2186 ts_tablet_manager.cc:1397] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.006s
I20260924 00:21:29.449211 2188 raft_consensus.cc:374] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.449548 2188 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.452291 2186 raft_consensus.cc:348] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.453341 2186 raft_consensus.cc:374] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.453701 2186 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.454550 2186 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.450260 2188 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.460736 2211 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.461328 2211 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.462167 1893 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:29.463366 2211 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.465349 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:29.466020 2142 raft_consensus.cc:2454] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:29.466936 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.468832 1960 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:21:29.469815 1960 leader_election.cc:341] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.470108 2186 ts_tablet_manager.cc:1428] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260924 00:21:29.470646 2211 raft_consensus.cc:2781] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.471037 2211 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.471527 2211 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.470940 2185 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.473656 2185 ts_tablet_manager.cc:1428] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.037s user 0.010s sys 0.001s
I20260924 00:21:29.474704 2185 tablet_bootstrap.cc:492] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.479002 2211 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.460723 2188 ts_tablet_manager.cc:1428] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.015s user 0.002s sys 0.004s
I20260924 00:21:29.481433 2211 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.481540 2185 tablet_bootstrap.cc:654] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.480635 2186 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:29.482952 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:29.483667 2142 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.486639 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:21:29.489004 1960 leader_election.cc:341] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.481665 2188 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.491961 2142 raft_consensus.cc:2454] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:29.493834 1960 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.494274 2185 tablet_bootstrap.cc:492] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.494751 2185 ts_tablet_manager.cc:1397] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.007s
I20260924 00:21:29.495044 2211 raft_consensus.cc:2781] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.495621 2211 raft_consensus.cc:686] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:29.497033 2211 consensus_queue.cc:237] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.497854 2185 raft_consensus.cc:348] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.498742 2185 raft_consensus.cc:374] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.499037 2185 raft_consensus.cc:729] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.499671 2185 consensus_queue.cc:260] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.502921 2185 ts_tablet_manager.cc:1428] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:21:29.503469 2188 tablet_bootstrap.cc:654] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.503996 2185 tablet_bootstrap.cc:492] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.510890 2185 tablet_bootstrap.cc:654] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.514505 2202 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.522442 2185 tablet_bootstrap.cc:492] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.523085 2185 ts_tablet_manager.cc:1397] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260924 00:21:29.524250 2186 tablet_bootstrap.cc:654] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.526492 2185 raft_consensus.cc:348] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.527316 1893 catalog_manager.cc:5495] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:29.528008 2185 raft_consensus.cc:374] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.528684 2185 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.529388 2185 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.538094 2185 ts_tablet_manager.cc:1428] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s
I20260924 00:21:29.539096 2185 tablet_bootstrap.cc:492] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.541082 2188 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.541666 2188 ts_tablet_manager.cc:1397] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.004s
I20260924 00:21:29.544878 2188 raft_consensus.cc:348] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.545714 2188 raft_consensus.cc:374] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.546121 2188 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.547319 2188 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.548162 2186 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:29.548727 2186 ts_tablet_manager.cc:1397] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.003s
I20260924 00:21:29.549921 2188 ts_tablet_manager.cc:1428] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260924 00:21:29.545809 2185 tablet_bootstrap.cc:654] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.545542 2261 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:29.550904 2188 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.554638 2186 raft_consensus.cc:348] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.555383 2186 raft_consensus.cc:374] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.555645 2186 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.556394 2186 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.559875 2186 ts_tablet_manager.cc:1428] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260924 00:21:29.560833 2186 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:29.561975 2273 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:29.566534 2273 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.567090 2273 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.571303 2273 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.571609 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.573876 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:29.576236 2109 leader_election.cc:341] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.576632 2185 tablet_bootstrap.cc:492] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.577119 2185 ts_tablet_manager.cc:1397] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20260924 00:21:29.574559 1995 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:29.577112 2188 tablet_bootstrap.cc:654] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.578912 2110 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.580487 2274 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.580843 2274 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.580782 2185 raft_consensus.cc:348] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.581346 2274 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.581665 2185 raft_consensus.cc:374] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.581959 2185 raft_consensus.cc:729] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.582578 2185 consensus_queue.cc:260] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.584771 2185 ts_tablet_manager.cc:1428] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260924 00:21:29.586061 2185 tablet_bootstrap.cc:492] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.586429 2071 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.586582 2186 tablet_bootstrap.cc:654] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.595981 2185 tablet_bootstrap.cc:654] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.601987 2260 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.602603 2260 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.604836 2260 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.609701 2273 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.611212 2273 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.623399 2281 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.624219 2281 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.633124 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.634599 1995 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:29.635830 2281 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.643832 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:29.650507 2035 leader_election.cc:341] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.642308 2260 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.634626 2274 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.664299 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:29.664927 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.665095 1995 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.666944 1994 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:29.668557 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.671087 2274 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:21:29.673197 2109 leader_election.cc:341] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.675393 2110 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.676096 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.676400 2231 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.676730 1995 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:29.678121 2231 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.677948 2110 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:29.678584 2231 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.678934 2269 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.678469 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:29.679118 2231 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.679461 2269 raft_consensus.cc:686] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:29.665997 2273 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.680545 2269 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:29.682785 2109 leader_election.cc:341] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.685117 2036 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:29.686470 2261 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.686853 2261 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.687183 2261 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:29.688928 1960 leader_election.cc:341] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.654166 2260 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.692414 2231 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.695853 2188 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.696417 2188 ts_tablet_manager.cc:1397] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.146s user 0.039s sys 0.039s
I20260924 00:21:29.695395 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:29.702432 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:21:29.704650 2109 leader_election.cc:341] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.711090 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:29.711911 1994 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.708027 2261 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:29.713766 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260924 00:21:29.699332 2317 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260924 00:21:29.720429 2261 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.720259 2231 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.734571 2190 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.736119 1994 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:29.735357 2190 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.763751 2190 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:29.721550 1960 leader_election.cc:341] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.764827 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:21:29.765988 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 44644.4us
I20260924 00:21:29.767122 2281 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.768939 2281 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.769411 2281 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.770506 2260 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.771268 2281 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.773284 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:29.774919 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:29.774616 1893 catalog_manager.cc:5495] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:29.775490 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.776206 1996 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:29.778996 2188 raft_consensus.cc:348] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.779735 2188 raft_consensus.cc:374] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.780004 2188 raft_consensus.cc:729] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.780587 2188 consensus_queue.cc:260] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.783210 2188 ts_tablet_manager.cc:1428] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.086s user 0.004s sys 0.007s
I20260924 00:21:29.784360 2188 tablet_bootstrap.cc:492] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.771044 2269 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:29.775523 1994 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.787525 2185 tablet_bootstrap.cc:492] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.788057 2185 ts_tablet_manager.cc:1397] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.202s user 0.012s sys 0.011s
I20260924 00:21:29.788587 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:29.789567 2036 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:29.790292 2186 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:29.790395 2261 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:29.791481 2261 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.791529 2185 raft_consensus.cc:348] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.795648 2185 raft_consensus.cc:374] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.787418 2110 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.796067 2185 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.795712 2261 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.796945 2185 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.790735 2186 ts_tablet_manager.cc:1397] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.230s user 0.037s sys 0.033s
I20260924 00:21:29.793009 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:29.795038 2035 leader_election.cc:341] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.802899 2185 ts_tablet_manager.cc:1428] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
I20260924 00:21:29.804034 2185 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.804740 1994 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:29.805603 2186 raft_consensus.cc:348] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.806303 2186 raft_consensus.cc:374] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.806727 2186 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.811224 2185 tablet_bootstrap.cc:654] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:29.814033 1960 leader_election.cc:341] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.814759 2324 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.815737 2324 raft_consensus.cc:686] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:29.816686 2324 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.807680 2186 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.819909 2186 ts_tablet_manager.cc:1428] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260924 00:21:29.820854 2186 tablet_bootstrap.cc:492] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:29.822453 2188 tablet_bootstrap.cc:654] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:29.824788 2035 leader_election.cc:341] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.825531 2036 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:29.826458 2202 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.826988 2202 raft_consensus.cc:686] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:29.828325 2202 consensus_queue.cc:237] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.830833 2186 tablet_bootstrap.cc:654] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.834859 2261 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.834733 2259 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.836725 2261 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.838912 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:29.839811 2144 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.845436 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:29.847750 1994 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.859084 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.861565 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:29.865008 1996 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.866935 2320 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.877614 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:29.882994 1960 leader_election.cc:341] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.883810 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.889494 2320 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.894436 2188 tablet_bootstrap.cc:492] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.894994 2188 ts_tablet_manager.cc:1397] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.111s user 0.021s sys 0.017s
I20260924 00:21:29.897715 2188 raft_consensus.cc:348] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.898502 2188 raft_consensus.cc:374] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.898797 2188 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.900126 2188 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.902491 2188 ts_tablet_manager.cc:1428] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260924 00:21:29.892584 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:29.903525 2188 tablet_bootstrap.cc:492] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.903944 2320 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:29.904321 1892 catalog_manager.cc:5495] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:29.906227 1994 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:29.911011 2185 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:29.911501 2185 ts_tablet_manager.cc:1397] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.108s user 0.024s sys 0.045s
I20260924 00:21:29.912708 2036 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:29.914321 2231 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.915127 2260 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:29.914875 2231 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:29.915656 2260 raft_consensus.cc:686] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:29.916647 2185 raft_consensus.cc:348] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.917030 2231 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:29.916531 2260 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.917409 2185 raft_consensus.cc:374] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.922267 2188 tablet_bootstrap.cc:654] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.923342 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:29.924139 1995 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:29.926276 2320 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:29.926741 2320 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.928773 2320 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:29.917699 2185 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.930786 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.931797 2185 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.934753 2188 tablet_bootstrap.cc:492] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:29.935269 2188 ts_tablet_manager.cc:1397] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.003s
W20260924 00:21:29.935815 1960 leader_election.cc:341] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:29.936075 2185 ts_tablet_manager.cc:1428] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20260924 00:21:29.937759 2185 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:29.938429 2188 raft_consensus.cc:348] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.939234 2188 raft_consensus.cc:374] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:29.939513 2188 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:29.940220 2188 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:29.943162 2188 ts_tablet_manager.cc:1428] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260924 00:21:29.946643 2185 tablet_bootstrap.cc:654] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.946496 2273 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:29.947638 2188 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:29.955425 2110 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:29.957527 2188 tablet_bootstrap.cc:654] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:29.957291 2144 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:29.969573 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:29.970547 2069 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:29.978209 2069 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:29.981990 1893 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:29.983295 2231 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:29.985459 2231 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:29.986019 2231 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:21:29.997007 2035 consensus_peers.cc:487] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Couldn't send request to peer 8c129c01e1c047628f6a87b001aa7258. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:29.995635 2071 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.002753 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.004364 2185 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.006413 2185 ts_tablet_manager.cc:1397] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.069s user 0.027s sys 0.003s
I20260924 00:21:30.012009 1892 catalog_manager.cc:5495] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:30.015789 2185 raft_consensus.cc:348] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.016942 2185 raft_consensus.cc:374] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.020390 2185 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.021119 2185 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.023312 2185 ts_tablet_manager.cc:1428] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260924 00:21:30.024160 2185 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:30.028609 2231 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:30.033968 2186 tablet_bootstrap.cc:492] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.034409 2186 ts_tablet_manager.cc:1397] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.214s user 0.035s sys 0.045s
W20260924 00:21:30.039361 1960 leader_election.cc:341] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.039925 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.041499 2190 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.054827 2188 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.052408 2231 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:30.055367 2188 ts_tablet_manager.cc:1397] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.108s user 0.013s sys 0.039s
I20260924 00:21:30.056927 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:30.057720 1994 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:30.059808 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:30.059428 2188 raft_consensus.cc:348] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.060793 2188 raft_consensus.cc:374] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.061178 2188 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.061800 2188 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.060613 2070 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:30.068966 2190 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:30.072173 2190 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.072630 2190 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.073745 2070 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:30.074891 2109 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:30.075829 2320 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.076153 2273 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:30.076860 2190 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.077534 2273 raft_consensus.cc:686] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:30.079888 2188 ts_tablet_manager.cc:1428] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.024s user 0.002s sys 0.005s
I20260924 00:21:30.080786 2188 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.081761 2273 consensus_queue.cc:237] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:30.084940 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:30.077549 1994 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:21:30.086627 1960 leader_election.cc:341] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.088812 2186 raft_consensus.cc:348] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.089776 2186 raft_consensus.cc:374] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.090126 2186 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.092049 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.092911 2142 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:30.101740 1960 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.099162 2186 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.103490 2190 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.106761 2186 ts_tablet_manager.cc:1428] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.072s user 0.007s sys 0.001s
I20260924 00:21:30.106798 2190 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.107666 2190 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.107973 2186 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.116541 2185 tablet_bootstrap.cc:654] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.126610 2190 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.131778 2211 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:30.136456 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:21:30.137760 1960 leader_election.cc:341] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.140297 2190 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.140841 2186 tablet_bootstrap.cc:654] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.144939 1891 catalog_manager.cc:5495] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 2, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:30.146514 2188 tablet_bootstrap.cc:654] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.148430 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:30.149046 2144 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.158080 2186 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.158615 2186 ts_tablet_manager.cc:1397] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.000s
I20260924 00:21:30.160725 2144 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:30.162858 1960 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.163960 2281 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:30.164290 2185 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.164417 2281 raft_consensus.cc:686] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:30.164981 2185 ts_tablet_manager.cc:1397] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.141s user 0.013s sys 0.003s
I20260924 00:21:30.165452 2281 consensus_queue.cc:237] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.170571 2186 raft_consensus.cc:348] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.170655 2185 raft_consensus.cc:348] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.171667 2185 raft_consensus.cc:374] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.172034 2185 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.171386 2186 raft_consensus.cc:374] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.172693 2185 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.173205 2186 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.174825 2190 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.174623 2186 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.175262 2185 ts_tablet_manager.cc:1428] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.010s user 0.003s sys 0.005s
I20260924 00:21:30.176330 2185 tablet_bootstrap.cc:492] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:30.175338 2190 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.178171 2186 ts_tablet_manager.cc:1428] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260924 00:21:30.178354 2190 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.179057 2186 tablet_bootstrap.cc:492] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.180181 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.180823 2144 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:30.181922 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:30.182623 2190 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.182973 2190 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.183269 2190 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.183238 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:30.184828 1960 leader_election.cc:341] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.188145 2211 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.188635 2211 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.190415 2211 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.191074 2190 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.193588 2188 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.193770 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.194075 2188 ts_tablet_manager.cc:1397] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.004s
I20260924 00:21:30.196123 2185 tablet_bootstrap.cc:654] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.196964 2188 raft_consensus.cc:348] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:30.196686 1960 leader_election.cc:341] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.197706 2188 raft_consensus.cc:374] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.197983 2188 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.198982 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:30.200337 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:30.200366 2188 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.201046 2142 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.201320 2070 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.205049 2202 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.205588 2202 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.208783 2202 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.208766 2070 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:30.209525 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.197059 2190 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.212105 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:30.213259 1994 raft_consensus.cc:2379] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:30.214627 2036 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:30.214823 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:30.215109 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:30.215831 2202 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:30.216259 1960 leader_election.cc:341] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.216805 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.217644 2211 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:30.217453 2142 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:30.217923 2190 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.218088 2211 raft_consensus.cc:686] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:30.219060 2211 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.223440 2142 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.213296 2188 ts_tablet_manager.cc:1428] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260924 00:21:30.227080 2188 tablet_bootstrap.cc:492] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.227438 2185 tablet_bootstrap.cc:492] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.232542 2188 tablet_bootstrap.cc:654] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.235771 2202 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.238436 2185 ts_tablet_manager.cc:1397] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.018s
I20260924 00:21:30.239281 2186 tablet_bootstrap.cc:654] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.241024 2185 raft_consensus.cc:348] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.241647 2185 raft_consensus.cc:374] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.241876 2185 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.242619 2185 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.243975 2188 tablet_bootstrap.cc:492] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.244441 2188 ts_tablet_manager.cc:1397] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260924 00:21:30.244645 2185 ts_tablet_manager.cc:1428] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260924 00:21:30.245481 2185 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:30.246429 2231 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.247354 2188 raft_consensus.cc:348] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.248137 2188 raft_consensus.cc:374] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.248413 2188 raft_consensus.cc:729] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.250766 2261 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:30.254987 2185 tablet_bootstrap.cc:654] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.255147 2186 tablet_bootstrap.cc:492] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.255656 2186 ts_tablet_manager.cc:1397] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.000s
I20260924 00:21:30.261795 2186 raft_consensus.cc:348] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.262527 2186 raft_consensus.cc:374] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.262784 2186 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.263379 2186 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.249033 2188 consensus_queue.cc:260] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.266602 2259 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.267035 2259 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.268648 2185 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.269093 2185 ts_tablet_manager.cc:1397] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.006s
I20260924 00:21:30.269032 2259 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.270718 1891 catalog_manager.cc:5495] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:30.272305 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:30.272959 1994 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:30.273324 2188 ts_tablet_manager.cc:1428] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.029s user 0.006s sys 0.002s
I20260924 00:21:30.274261 2188 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.274580 2036 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:30.275781 2202 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.276109 2202 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.276433 2202 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.278600 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.279290 2144 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:30.280560 2188 tablet_bootstrap.cc:654] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.282702 2261 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.288102 2202 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.290746 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:30.290652 2185 raft_consensus.cc:348] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.291432 1996 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.292109 2185 raft_consensus.cc:374] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.292598 2185 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.293287 2185 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.298797 2269 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:30.299485 1996 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:30.300589 2036 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:30.301925 2261 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:30.302390 2261 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:30.303211 2261 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.305262 2186 ts_tablet_manager.cc:1428] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.049s user 0.009s sys 0.000s
I20260924 00:21:30.304648 2273 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:30.306442 2186 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.308372 2202 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.312054 2185 ts_tablet_manager.cc:1428] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.043s user 0.003s sys 0.003s
I20260924 00:21:30.313107 2185 tablet_bootstrap.cc:492] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:30.316737 2186 tablet_bootstrap.cc:654] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.319427 2188 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.319470 2185 tablet_bootstrap.cc:654] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.320094 2188 ts_tablet_manager.cc:1397] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.016s
I20260924 00:21:30.323141 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:30.323805 2144 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.335912 2188 raft_consensus.cc:348] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.336711 2188 raft_consensus.cc:374] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.336966 2188 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.337608 2188 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.339916 2188 ts_tablet_manager.cc:1428] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.019s user 0.004s sys 0.004s
I20260924 00:21:30.340812 2188 tablet_bootstrap.cc:492] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.346987 2188 tablet_bootstrap.cc:654] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.348167 2070 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.360770 2281 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.361387 2281 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.363569 2281 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.365084 2186 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.366137 2186 ts_tablet_manager.cc:1397] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.005s
I20260924 00:21:30.365262 2144 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:30.369370 2186 raft_consensus.cc:348] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.370478 2186 raft_consensus.cc:374] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.371052 2186 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.370594 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.371807 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:30.377914 1960 leader_election.cc:341] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:30.378577 1960 leader_election.cc:341] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.378953 1960 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.372285 2273 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.380565 2281 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.384090 2185 tablet_bootstrap.cc:492] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.384369 2186 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.384873 2185 ts_tablet_manager.cc:1397] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.072s user 0.019s sys 0.015s
I20260924 00:21:30.388747 2211 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.391053 2186 ts_tablet_manager.cc:1428] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.025s user 0.001s sys 0.008s
I20260924 00:21:30.389426 2211 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.391841 2185 raft_consensus.cc:348] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.393224 2185 raft_consensus.cc:374] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.393474 2186 tablet_bootstrap.cc:492] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.393617 2185 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.394027 2211 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.394567 2185 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.396111 2188 tablet_bootstrap.cc:492] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.396306 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:30.396550 2188 ts_tablet_manager.cc:1397] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.009s
I20260924 00:21:30.397686 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:30.398228 1960 leader_election.cc:341] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.398697 2185 ts_tablet_manager.cc:1428] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
W20260924 00:21:30.399248 1960 leader_election.cc:341] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.399616 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.399616 2185 tablet_bootstrap.cc:492] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:30.403113 2211 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.405799 2188 raft_consensus.cc:348] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.406788 2188 raft_consensus.cc:374] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.406396 1894 catalog_manager.cc:5495] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:30.407135 2188 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.408075 2188 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.411440 2188 ts_tablet_manager.cc:1428] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260924 00:21:30.412648 2188 tablet_bootstrap.cc:492] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.414390 2260 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.419338 2144 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.424021 2186 tablet_bootstrap.cc:654] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.424921 2188 tablet_bootstrap.cc:654] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.424475 1891 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:30.446085 2185 tablet_bootstrap.cc:654] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.444051 2261 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.447270 2261 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.451610 2211 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.452149 2211 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.453944 2211 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.454154 2261 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.456465 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:30.457293 1995 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:30.457531 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.458899 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:30.459036 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:30.458912 2035 leader_election.cc:341] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:30.461273 1960 leader_election.cc:341] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:30.461999 1960 leader_election.cc:341] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.462354 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.463287 2186 tablet_bootstrap.cc:492] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.463855 2186 ts_tablet_manager.cc:1397] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.002s
I20260924 00:21:30.465005 2211 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.466781 2036 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:30.469861 2261 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.470273 2261 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.470600 2261 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.482057 2186 raft_consensus.cc:348] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.482951 2186 raft_consensus.cc:374] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.483227 2186 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.483808 2186 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.486277 2261 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.488867 2259 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.489641 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:30.490566 1995 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.494306 2261 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.499647 2186 ts_tablet_manager.cc:1428] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.035s user 0.005s sys 0.003s
I20260924 00:21:30.501523 2186 tablet_bootstrap.cc:492] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.511360 2186 tablet_bootstrap.cc:654] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.511164 1995 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:30.521179 2188 tablet_bootstrap.cc:492] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.521098 2036 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:30.521826 2188 ts_tablet_manager.cc:1397] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.109s user 0.047s sys 0.025s
I20260924 00:21:30.522454 2261 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:30.523519 2261 raft_consensus.cc:686] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:30.525545 2261 consensus_queue.cc:237] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.528179 2188 raft_consensus.cc:348] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.528934 2188 raft_consensus.cc:374] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.529268 2188 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.530145 2188 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.533610 2188 ts_tablet_manager.cc:1428] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260924 00:21:30.534804 2188 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.539727 2211 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.540419 2211 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.543263 2211 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.545922 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.545280 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:30.549831 1960 leader_election.cc:341] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.544003 2188 tablet_bootstrap.cc:654] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:30.552634 1960 leader_election.cc:341] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.553620 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:30.558907 2281 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.564895 2188 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.565392 2188 ts_tablet_manager.cc:1397] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.011s
I20260924 00:21:30.567307 2188 raft_consensus.cc:348] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.567804 2188 raft_consensus.cc:374] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.568030 2188 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.568553 2188 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.574379 2188 ts_tablet_manager.cc:1428] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260924 00:21:30.575644 2188 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.577502 2185 tablet_bootstrap.cc:492] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.578119 2185 ts_tablet_manager.cc:1397] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.179s user 0.017s sys 0.009s
I20260924 00:21:30.581513 2185 raft_consensus.cc:348] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.582746 2185 raft_consensus.cc:374] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.583105 2185 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.592235 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:30.594525 2188 tablet_bootstrap.cc:654] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:30.604264 2035 leader_election.cc:341] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.610630 1994 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260924 00:21:30.618827 2259 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.624784 2186 tablet_bootstrap.cc:492] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.625478 2259 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.626427 2186 ts_tablet_manager.cc:1397] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.126s user 0.030s sys 0.067s
I20260924 00:21:30.628001 2259 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.628927 2320 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.630414 2320 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.631348 2186 raft_consensus.cc:348] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.632035 2186 raft_consensus.cc:374] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.632278 2186 raft_consensus.cc:729] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.632944 2320 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.632874 2186 consensus_queue.cc:260] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.637455 2186 ts_tablet_manager.cc:1428] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20260924 00:21:30.637518 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.638348 2186 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.638449 2144 raft_consensus.cc:2379] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:30.639609 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.637519 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:30.643114 2070 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.643451 2273 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.644966 2186 tablet_bootstrap.cc:654] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.646545 2185 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.648623 2185 ts_tablet_manager.cc:1428] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.070s user 0.009s sys 0.003s
I20260924 00:21:30.648746 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:30.649734 2185 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:30.650027 2071 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:30.650426 1894 catalog_manager.cc:5495] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:30.658506 2185 tablet_bootstrap.cc:654] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.659179 2071 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260924 00:21:30.659370 2186 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.660812 2036 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:30.660988 2273 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.662056 2259 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:30.663153 1960 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:30.666965 2186 ts_tablet_manager.cc:1397] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.009s
I20260924 00:21:30.670179 2186 raft_consensus.cc:348] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.670635 2185 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.670918 2186 raft_consensus.cc:374] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.671198 2185 ts_tablet_manager.cc:1397] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.009s
I20260924 00:21:30.671223 2186 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.672257 2186 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.674674 2320 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.675830 2320 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.676431 2320 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.674779 2186 ts_tablet_manager.cc:1428] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260924 00:21:30.678202 2186 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
W20260924 00:21:30.680923 1960 leader_election.cc:341] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.684589 2281 consensus_queue.cc:1035] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260924 00:21:30.691499 2261 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.693100 2185 raft_consensus.cc:348] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.693990 2185 raft_consensus.cc:374] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.694165 2188 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.694342 2185 raft_consensus.cc:729] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.694692 2188 ts_tablet_manager.cc:1397] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.004s
I20260924 00:21:30.695123 2185 consensus_queue.cc:260] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.692449 2261 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.698568 2261 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.699687 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:30.701354 2035 leader_election.cc:341] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.702334 2185 ts_tablet_manager.cc:1428] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260924 00:21:30.703299 2185 tablet_bootstrap.cc:492] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:30.703729 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:30.705322 2281 consensus_queue.cc:1035] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:30.704448 1995 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:30.708374 2188 raft_consensus.cc:348] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.710009 2188 raft_consensus.cc:374] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.710273 2188 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.713790 2188 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.714284 2036 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:30.715129 2261 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.715507 2261 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.715832 2261 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.722419 2188 ts_tablet_manager.cc:1428] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260924 00:21:30.723418 2188 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:30.725809 2185 tablet_bootstrap.cc:654] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.699649 2320 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.733484 2320 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.735688 2261 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.739521 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:30.741513 2069 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.743304 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:21:30.746522 1960 leader_election.cc:341] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.746147 2261 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.748956 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:30.749651 1995 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.751031 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:21:30.752807 2035 leader_election.cc:341] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.758059 2186 tablet_bootstrap.cc:654] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.759814 2069 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:30.779058 1960 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:30.779970 2190 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:30.789326 2281 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:30.789863 2190 raft_consensus.cc:686] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:30.790874 2190 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.796895 2320 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:30.797969 1995 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:30.801496 2188 tablet_bootstrap.cc:654] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.802665 2186 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.805817 2186 ts_tablet_manager.cc:1397] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.128s user 0.005s sys 0.009s
I20260924 00:21:30.811149 2036 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:30.812150 2261 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:30.814205 2261 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:30.815086 2261 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.823243 2186 raft_consensus.cc:348] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.833173 2186 raft_consensus.cc:374] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.835034 2186 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.836616 2186 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.825940 2259 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.853549 2186 ts_tablet_manager.cc:1428] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.046s user 0.010s sys 0.000s
I20260924 00:21:30.854401 2186 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.828438 2202 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260924 00:21:30.857345 2185 tablet_bootstrap.cc:492] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.857968 2185 ts_tablet_manager.cc:1397] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.155s user 0.037s sys 0.044s
I20260924 00:21:30.850981 2259 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.887202 2259 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:30.897202 2185 raft_consensus.cc:348] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.897879 2185 raft_consensus.cc:374] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.898137 2185 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.898722 2185 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.907809 2185 ts_tablet_manager.cc:1428] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.049s user 0.007s sys 0.001s
I20260924 00:21:30.909379 2185 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
W20260924 00:21:30.911909 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:21:30.929754 2186 tablet_bootstrap.cc:654] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.930039 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:30.932991 1894 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:21:30.935066 2035 leader_election.cc:341] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:30.942979 2185 tablet_bootstrap.cc:654] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:30.955883 2188 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:30.956316 2188 ts_tablet_manager.cc:1397] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.233s user 0.009s sys 0.004s
I20260924 00:21:30.956674 2186 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:30.957222 2186 ts_tablet_manager.cc:1397] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.103s user 0.011s sys 0.002s
I20260924 00:21:30.956698 2185 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:30.957708 2185 ts_tablet_manager.cc:1397] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.011s
I20260924 00:21:30.959088 2188 raft_consensus.cc:348] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.959825 2188 raft_consensus.cc:374] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.960192 2188 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.960144 2185 raft_consensus.cc:348] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.961086 2185 raft_consensus.cc:374] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.961424 2185 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.961776 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:30.962150 2274 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.962589 2274 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.962721 2185 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.964196 2185 ts_tablet_manager.cc:1428] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 00:21:30.964526 2274 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:30.962586 1995 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:30.965914 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:30.966540 1994 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:30.967669 2110 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:30.959236 2186 raft_consensus.cc:348] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.968487 2274 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.968567 2186 raft_consensus.cc:374] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:30.968863 2274 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.969082 2186 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:30.969300 2274 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.970278 2186 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.973826 2186 ts_tablet_manager.cc:1428] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260924 00:21:30.975090 2186 tablet_bootstrap.cc:492] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:30.976424 2274 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.980449 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:30.981191 1994 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.982997 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:30.985641 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:30.988412 2384 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:30.988794 2384 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:30.989218 2384 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:30.997596 2320 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.998148 2320 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:30.992930 2188 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.000554 2320 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.002631 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.002884 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.004972 2384 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.007018 2188 ts_tablet_manager.cc:1428] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.050s user 0.007s sys 0.002s
I20260924 00:21:31.011672 2188 tablet_bootstrap.cc:492] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.016238 1893 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:21:31.017392 1960 leader_election.cc:341] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.017490 2385 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:21:31.018266 2109 leader_election.cc:341] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.018591 2186 tablet_bootstrap.cc:654] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:31.018181 1960 leader_election.cc:341] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.019441 1960 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.020336 2211 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.022830 2188 tablet_bootstrap.cc:654] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.024770 1994 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:31.025372 2269 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:30.977782 2185 tablet_bootstrap.cc:492] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:31.025971 2110 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:31.025906 2269 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.027379 2273 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.027859 2273 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:31.028475 2269 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.028749 2273 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.031684 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.032465 1995 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:31.032835 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.033567 2070 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:31.034015 2110 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:31.035641 2324 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.036049 2324 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.036381 2324 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.037348 2259 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:31.041833 2274 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.042507 2384 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.045686 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:31.048623 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.049269 1995 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.055652 2185 tablet_bootstrap.cc:654] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:31.049453 2109 leader_election.cc:341] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.059810 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:31.060496 2145 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.058570 2324 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.066955 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.067620 1994 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.069466 2388 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.070361 2388 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.071674 2188 tablet_bootstrap.cc:492] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.073501 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.073815 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:31.075078 1960 leader_election.cc:341] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.075518 2188 ts_tablet_manager.cc:1397] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.068s user 0.021s sys 0.008s
W20260924 00:21:31.075737 1960 leader_election.cc:341] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.076141 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.073956 2324 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.077279 2211 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.073565 2388 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.079780 1994 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:31.080531 1995 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.080948 2110 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:31.081630 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.081806 2188 raft_consensus.cc:348] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.082439 2384 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.082861 2188 raft_consensus.cc:374] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.083314 2188 raft_consensus.cc:729] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.083379 2384 raft_consensus.cc:686] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:31.084376 2384 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.084407 2188 consensus_queue.cc:260] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.082554 2273 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.088186 2273 raft_consensus.cc:686] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:31.088841 2188 ts_tablet_manager.cc:1428] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
I20260924 00:21:31.089334 2273 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.090296 2188 tablet_bootstrap.cc:492] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.067569 2145 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.097808 2324 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.098353 2324 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.098399 1894 catalog_manager.cc:5495] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:31.097656 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:31.101457 2070 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.105808 2188 tablet_bootstrap.cc:654] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.106894 2185 tablet_bootstrap.cc:492] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:31.107347 2185 ts_tablet_manager.cc:1397] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.130s user 0.021s sys 0.027s
I20260924 00:21:31.110718 2185 raft_consensus.cc:348] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.111428 2185 raft_consensus.cc:374] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.111696 2185 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.112392 2185 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.114677 2185 ts_tablet_manager.cc:1428] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:21:31.115568 2185 tablet_bootstrap.cc:492] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:31.121728 2185 tablet_bootstrap.cc:654] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.122277 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.123608 1995 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:31.124864 2110 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:31.126003 2070 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:31.126139 2274 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.100978 2324 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.127226 2324 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.126770 2274 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.127569 2324 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.127907 2324 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.129750 2274 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.131739 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.134004 2186 tablet_bootstrap.cc:492] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.135532 2186 ts_tablet_manager.cc:1397] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.161s user 0.017s sys 0.006s
I20260924 00:21:31.136953 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.138096 2070 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:31.141741 2186 raft_consensus.cc:348] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.142989 2186 raft_consensus.cc:374] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.143293 2186 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.143959 2186 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.137593 2324 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.146162 2186 ts_tablet_manager.cc:1428] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260924 00:21:31.147079 2186 tablet_bootstrap.cc:492] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.147418 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.148056 1995 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.148849 2188 tablet_bootstrap.cc:492] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.149322 2188 ts_tablet_manager.cc:1397] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.024s
I20260924 00:21:31.152070 2388 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.152328 2188 raft_consensus.cc:348] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.152602 2388 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.153064 2188 raft_consensus.cc:374] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.153416 2188 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.153822 2186 tablet_bootstrap.cc:654] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.154085 2188 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.154633 2388 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.154912 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.155743 2070 raft_consensus.cc:2379] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:31.156198 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.156430 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:31.157503 1960 leader_election.cc:341] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:31.158160 1960 leader_election.cc:341] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.158496 2188 ts_tablet_manager.cc:1428] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260924 00:21:31.158551 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.159449 2188 tablet_bootstrap.cc:492] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.160163 2388 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.162817 2324 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.164875 1995 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:31.165877 2109 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.166203 2188 tablet_bootstrap.cc:654] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.166568 2110 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:31.167068 2273 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.167550 2274 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.168067 2274 raft_consensus.cc:686] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:31.168923 2274 consensus_queue.cc:237] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.179224 2274 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.179769 2274 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.182581 2274 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.183439 2273 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.184590 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.185700 1995 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:31.187446 2110 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:31.188890 2188 tablet_bootstrap.cc:492] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.189851 2188 ts_tablet_manager.cc:1397] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.005s
I20260924 00:21:31.193609 2186 tablet_bootstrap.cc:492] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.193192 2188 raft_consensus.cc:348] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.194041 2188 raft_consensus.cc:374] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.194317 2186 ts_tablet_manager.cc:1397] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.000s
I20260924 00:21:31.194335 2188 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.195365 2324 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.195255 2188 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.195997 2324 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.197985 2188 ts_tablet_manager.cc:1428] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:21:31.190704 1891 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:31.199004 2188 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
W20260924 00:21:31.200567 2035 consensus_peers.cc:487] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Couldn't send request to peer 8c129c01e1c047628f6a87b001aa7258. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:31.197518 2186 raft_consensus.cc:348] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.201370 2186 raft_consensus.cc:374] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.201759 2186 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.203233 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:31.205960 2069 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.206049 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:31.210757 2109 leader_election.cc:341] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.212984 2188 tablet_bootstrap.cc:654] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.217726 2388 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:31.219812 2069 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:31.196537 2324 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.220906 2186 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.229311 2185 tablet_bootstrap.cc:492] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:31.231133 2185 ts_tablet_manager.cc:1397] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.116s user 0.039s sys 0.010s
I20260924 00:21:31.231621 2186 ts_tablet_manager.cc:1428] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20260924 00:21:31.232863 2186 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.234344 2185 raft_consensus.cc:348] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.234967 2185 raft_consensus.cc:374] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.235216 2185 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.235822 2185 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.237882 2185 ts_tablet_manager.cc:1428] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 00:21:31.238868 2185 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:31.240767 2324 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.245627 2388 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.246299 2388 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.249915 2185 tablet_bootstrap.cc:654] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.254587 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.255537 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:31.256462 1960 leader_election.cc:341] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.256762 2186 tablet_bootstrap.cc:654] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.249946 2388 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:31.261519 1960 leader_election.cc:341] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.261958 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.262970 2324 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.263100 2388 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.264583 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:31.265884 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.266770 1995 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:31.268600 2109 leader_election.cc:341] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.273518 2188 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.273993 2188 ts_tablet_manager.cc:1397] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.008s
I20260924 00:21:31.276629 2188 raft_consensus.cc:348] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.277350 2188 raft_consensus.cc:374] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.277603 2188 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.282764 1995 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:31.286149 2110 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.287282 2273 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.288007 2273 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:31.289215 2273 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.293217 1891 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:31.295390 1891 catalog_manager.cc:5495] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:31.278209 2188 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.302439 2186 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.302956 2186 ts_tablet_manager.cc:1397] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.070s user 0.020s sys 0.015s
I20260924 00:21:31.307695 2186 raft_consensus.cc:348] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.308745 2186 raft_consensus.cc:374] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.309078 2186 raft_consensus.cc:729] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.309758 2186 consensus_queue.cc:260] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.311815 2186 ts_tablet_manager.cc:1428] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260924 00:21:31.312633 2186 tablet_bootstrap.cc:492] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.312665 2281 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:31.314865 2320 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.315297 2320 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.317483 2320 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.319119 2188 ts_tablet_manager.cc:1428] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20260924 00:21:31.322263 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.326974 2186 tablet_bootstrap.cc:654] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.327028 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:31.329775 1960 leader_election.cc:341] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:31.331151 1960 leader_election.cc:341] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.332013 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.332710 2388 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.334237 2384 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:31.336109 2384 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.336619 2384 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.339053 2384 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.341210 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.342181 1994 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:31.343214 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:31.344772 2035 leader_election.cc:341] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.346218 2036 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:31.346978 2188 tablet_bootstrap.cc:492] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.351799 2185 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:31.352937 2384 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.353799 2384 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.357862 2384 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.374884 2188 tablet_bootstrap.cc:654] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.375336 2186 tablet_bootstrap.cc:492] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.375737 2186 ts_tablet_manager.cc:1397] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.063s user 0.039s sys 0.014s
I20260924 00:21:31.352960 2185 ts_tablet_manager.cc:1397] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.114s user 0.009s sys 0.025s
I20260924 00:21:31.377929 2186 raft_consensus.cc:348] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.378690 2186 raft_consensus.cc:374] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.379029 2186 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.380003 2324 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.379786 2186 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.380443 2324 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.381647 2186 ts_tablet_manager.cc:1428] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260924 00:21:31.382360 2186 tablet_bootstrap.cc:492] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.382805 2324 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.383877 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.385103 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.386821 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.387995 2186 tablet_bootstrap.cc:654] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.389417 2188 tablet_bootstrap.cc:492] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.387990 1891 catalog_manager.cc:5495] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:31.389909 2188 ts_tablet_manager.cc:1397] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.001s
I20260924 00:21:31.392478 2188 raft_consensus.cc:348] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.393245 2188 raft_consensus.cc:374] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.393532 2188 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.395321 2324 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.396055 2384 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.399688 2186 tablet_bootstrap.cc:492] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.400154 2186 ts_tablet_manager.cc:1397] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.015s
I20260924 00:21:31.384688 2185 raft_consensus.cc:348] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.401546 2185 raft_consensus.cc:374] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.401865 2185 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.402676 2185 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.403434 2186 raft_consensus.cc:348] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.403937 2145 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.404129 2186 raft_consensus.cc:374] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.404452 2186 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.404889 2185 ts_tablet_manager.cc:1428] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20260924 00:21:31.405095 2186 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.405874 2185 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:31.412500 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:31.413043 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.413673 1995 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:31.414273 2035 leader_election.cc:341] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.420418 1995 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.407246 2186 ts_tablet_manager.cc:1428] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260924 00:21:31.422065 2186 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.408375 2384 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.421825 2036 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:31.394141 2188 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.423719 2259 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:31.424962 2259 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.425874 2259 raft_consensus.cc:686] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:31.426649 2259 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.427753 2188 ts_tablet_manager.cc:1428] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.038s user 0.003s sys 0.004s
I20260924 00:21:31.428027 2186 tablet_bootstrap.cc:654] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.428660 2188 tablet_bootstrap.cc:492] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.435142 2202 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.436000 2188 tablet_bootstrap.cc:654] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.435664 2202 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.441979 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.442727 1994 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:31.444690 2036 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.446307 2261 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.446679 2261 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.447057 2261 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.447793 2188 tablet_bootstrap.cc:492] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.448251 2188 ts_tablet_manager.cc:1397] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20260924 00:21:31.451529 2188 raft_consensus.cc:348] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.452206 2188 raft_consensus.cc:374] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.452474 2188 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.439215 2202 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.453174 2188 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.464639 1891 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:31.468068 2186 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.468565 2186 ts_tablet_manager.cc:1397] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.028s
I20260924 00:21:31.467864 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.469108 2142 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:31.468976 2261 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.470523 2188 ts_tablet_manager.cc:1428] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
I20260924 00:21:31.470954 2186 raft_consensus.cc:348] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.471614 2186 raft_consensus.cc:374] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.471961 2186 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.471604 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.472592 1993 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.472779 2188 tablet_bootstrap.cc:492] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.479116 2185 tablet_bootstrap.cc:654] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.480047 1993 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.472570 2186 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.481753 2036 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.483374 2186 ts_tablet_manager.cc:1428] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260924 00:21:31.483924 2261 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.484274 2186 tablet_bootstrap.cc:492] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.480233 2188 tablet_bootstrap.cc:654] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.484972 2261 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.485618 2261 raft_consensus.cc:686] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:31.486897 2070 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.491959 2188 tablet_bootstrap.cc:492] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.492679 2188 ts_tablet_manager.cc:1397] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.002s
I20260924 00:21:31.494746 2186 tablet_bootstrap.cc:654] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.496018 2188 raft_consensus.cc:348] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.496939 2188 raft_consensus.cc:374] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.497304 2188 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.497977 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:31.498839 2188 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.506613 2142 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.507570 2273 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:31.509395 2185 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:31.509878 2185 ts_tablet_manager.cc:1397] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.104s user 0.016s sys 0.007s
I20260924 00:21:31.513620 2185 raft_consensus.cc:348] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.514364 2185 raft_consensus.cc:374] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.514109 2188 ts_tablet_manager.cc:1428] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 00:21:31.514654 2185 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.515818 2185 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.516263 2188 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.517997 2185 ts_tablet_manager.cc:1428] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:21:31.518841 2185 tablet_bootstrap.cc:492] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:31.523132 2273 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.523856 2273 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.486594 2261 consensus_queue.cc:237] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.528599 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.529577 1994 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:31.532485 2110 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:31.533380 2385 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.534029 2185 tablet_bootstrap.cc:654] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.533860 2385 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.534843 2324 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.536283 2324 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.536633 2324 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.537302 2385 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.543401 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.539902 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.551519 1994 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:31.553220 2259 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.553714 2259 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.560683 2274 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260924 00:21:31.552906 2142 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.563268 2385 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:31.574008 2035 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:31.576042 2383 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.569365 2274 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260924 00:21:31.578009 2186 tablet_bootstrap.cc:492] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.578585 2186 ts_tablet_manager.cc:1397] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.094s user 0.042s sys 0.042s
I20260924 00:21:31.575131 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.581561 1993 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.588011 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.588318 2273 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.589211 2384 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.589339 2259 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.589684 2384 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.590261 2259 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:31.590593 2259 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.590727 2388 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.590939 2259 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:31.592149 2384 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.585551 1891 catalog_manager.cc:5495] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:31.591509 2388 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.608520 2186 raft_consensus.cc:348] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.614739 2186 raft_consensus.cc:374] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.615319 2186 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.616086 2186 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.622686 2185 tablet_bootstrap.cc:492] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:31.623250 2185 ts_tablet_manager.cc:1397] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.105s user 0.018s sys 0.006s
I20260924 00:21:31.627490 2185 raft_consensus.cc:348] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.628201 2185 raft_consensus.cc:374] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.628468 2185 raft_consensus.cc:729] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.630573 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.631524 2186 ts_tablet_manager.cc:1428] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.053s user 0.004s sys 0.004s
I20260924 00:21:31.597306 2324 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.632543 2188 tablet_bootstrap.cc:654] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.632639 2186 tablet_bootstrap.cc:492] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.629065 2185 consensus_queue.cc:260] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.629923 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.635713 1993 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:31.637431 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.640748 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.641180 2036 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.641616 2145 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:31.642047 2385 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.642390 2385 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.642743 2385 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.645968 2185 ts_tablet_manager.cc:1428] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20260924 00:21:31.646901 2388 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.637478 2324 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.648844 2185 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:31.650470 2142 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.652747 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:31.654458 1960 leader_election.cc:341] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.660844 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.660605 2259 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.661602 1993 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.662385 2071 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260924 00:21:31.671869 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.672523 1995 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:31.687155 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:31.691006 1995 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:31.694005 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.695127 2261 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:31.695310 2259 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.695865 2261 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
W20260924 00:21:31.698284 2109 leader_election.cc:341] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.697826 2261 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.703013 2269 consensus_queue.cc:1046] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:31.705859 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.716203 2259 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260924 00:21:31.719884 2185 tablet_bootstrap.cc:654] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.712008 2383 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.723488 2423 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:31.725267 2423 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:31.723253 2324 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260924 00:21:31.712565 2109 leader_election.cc:341] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.728049 2186 tablet_bootstrap.cc:654] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.727195 2385 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.751104 2188 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:31.751585 2188 ts_tablet_manager.cc:1397] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.236s user 0.008s sys 0.008s
I20260924 00:21:31.741755 2273 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.754163 2273 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.756341 2273 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:21:31.753346 1960 leader_election.cc:341] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.757285 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.758651 2388 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.759615 2259 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.759816 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.760507 1994 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.760167 2259 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.761953 2188 raft_consensus.cc:348] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.762614 2188 raft_consensus.cc:374] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.762933 2188 raft_consensus.cc:729] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.764253 2188 consensus_queue.cc:260] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.767485 2188 ts_tablet_manager.cc:1428] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260924 00:21:31.767306 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.767624 2385 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.768550 2188 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:31.769430 1996 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:31.770789 2036 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.762982 2259 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.775669 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:31.771710 2385 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:31.777428 2385 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:31.789529 1993 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:31.787743 2274 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:31.780434 2385 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.792927 1991 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260924 00:21:31.794697 2185 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:31.795361 2185 ts_tablet_manager.cc:1397] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.147s user 0.017s sys 0.014s
I20260924 00:21:31.798563 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:31.802011 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.802368 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:31.802948 1992 raft_consensus.cc:2379] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:31.790939 2274 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.802073 2141 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.804709 2320 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:31.820854 2185 raft_consensus.cc:348] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.823060 2185 raft_consensus.cc:374] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 00:21:31.817395 2035 leader_election.cc:341] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.819388 2110 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.823763 2185 raft_consensus.cc:729] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.834107 2185 consensus_queue.cc:260] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.836649 2385 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.837812 2185 ts_tablet_manager.cc:1428] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.042s user 0.009s sys 0.004s
I20260924 00:21:31.839080 2185 tablet_bootstrap.cc:492] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:31.839408 2385 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.845999 2186 tablet_bootstrap.cc:492] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:31.846577 2186 ts_tablet_manager.cc:1397] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.214s user 0.015s sys 0.021s
I20260924 00:21:31.821015 1994 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.846565 1891 catalog_manager.cc:5495] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 2, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 2 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:31.825851 2320 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.854396 2274 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:31.857767 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.859192 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:31.860342 1960 leader_election.cc:341] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:31.860948 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 44716.4us
I20260924 00:21:31.862582 2273 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:31.863283 2273 raft_consensus.cc:686] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:31.864131 2273 consensus_queue.cc:237] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.812862 2423 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.867479 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:31.868168 1991 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.857101 2320 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.895632 2423 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:21:31.898798 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 79065.8us
I20260924 00:21:31.905947 2110 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:31.906790 2324 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:31.906396 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:31.898674 2388 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:21:31.912222 2035 leader_election.cc:341] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.916165 2186 raft_consensus.cc:348] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.917047 2186 raft_consensus.cc:374] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:31.917382 2186 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:31.918056 2186 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.910512 2388 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.924089 2388 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:31.924644 2185 tablet_bootstrap.cc:654] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.929514 2186 ts_tablet_manager.cc:1428] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.082s user 0.007s sys 0.001s
I20260924 00:21:31.930840 2036 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:31.943765 2186 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:31.943990 2188 tablet_bootstrap.cc:654] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:31.926817 2274 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260924 00:21:31.948246 1894 catalog_manager.cc:5495] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:31.959039 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:31.955644 2383 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
W20260924 00:21:31.965859 1960 leader_election.cc:341] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:31.969578 2383 raft_consensus.cc:686] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:31.978271 2383 consensus_queue.cc:237] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:31.978915 2259 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:31.982761 2324 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:31.976562 2141 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.987957 2144 raft_consensus.cc:2396] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 00:21:31.970580 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:31.995645 2423 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260924 00:21:32.005983 1991 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:31.994733 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.010387 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:32.028832 1960 leader_election.cc:341] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.029448 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.030397 2211 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.007634 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.053107 2186 tablet_bootstrap.cc:654] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:32.081812 1960 leader_election.cc:341] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.082490 1960 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.088999 2211 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.095503 2185 tablet_bootstrap.cc:492] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.095992 2185 ts_tablet_manager.cc:1397] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.257s user 0.020s sys 0.007s
I20260924 00:21:32.099876 2188 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:32.100327 2188 ts_tablet_manager.cc:1397] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.332s user 0.026s sys 0.050s
I20260924 00:21:32.108865 2185 raft_consensus.cc:348] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.109750 2185 raft_consensus.cc:374] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.110131 2185 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.110735 2185 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.113941 2324 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260924 00:21:32.101595 2036 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:32.117157 2211 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.117669 2211 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.120496 2185 ts_tablet_manager.cc:1428] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20260924 00:21:32.122041 2185 tablet_bootstrap.cc:492] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.129333 2281 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.129884 2281 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.132081 2281 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.132946 2211 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.120110 2441 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:32.141207 2441 raft_consensus.cc:686] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:32.143126 2109 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.147080 2273 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260924 00:21:32.149539 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:32.151568 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:32.146266 1891 catalog_manager.cc:5495] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:32.156392 2385 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:32.153282 2324 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 00:21:32.159603 1960 leader_election.cc:341] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:32.160265 1960 leader_election.cc:341] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.142212 2441 consensus_queue.cc:237] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.182611 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.183529 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:32.187345 1960 leader_election.cc:341] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.187821 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.187563 2188 raft_consensus.cc:348] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.188683 2188 raft_consensus.cc:374] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.189023 2188 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.192502 2188 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.193642 2186 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.194146 2186 ts_tablet_manager.cc:1397] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.251s user 0.019s sys 0.015s
I20260924 00:21:32.190200 2388 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.194758 2188 ts_tablet_manager.cc:1428] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.094s user 0.006s sys 0.001s
I20260924 00:21:32.196164 2188 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
W20260924 00:21:32.198602 1960 leader_election.cc:341] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.199204 1960 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.202224 2211 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.211050 2186 raft_consensus.cc:348] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.211833 2186 raft_consensus.cc:374] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.212088 2186 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.212699 2186 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:32.215440 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:21:32.219751 2384 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:32.219192 2441 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.221580 2441 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.223580 2441 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.224874 2185 tablet_bootstrap.cc:654] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.228209 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.234774 2383 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:32.242215 2186 ts_tablet_manager.cc:1428] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20260924 00:21:32.243135 2186 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.248809 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:32.250164 1992 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260924 00:21:32.250373 1993 raft_consensus.cc:2379] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:32.261217 2383 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260924 00:21:32.269537 2036 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:32.273447 2186 tablet_bootstrap.cc:654] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.272723 1894 catalog_manager.cc:5495] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:32.272208 2383 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.276875 2185 tablet_bootstrap.cc:492] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.277334 2185 ts_tablet_manager.cc:1397] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.006s
I20260924 00:21:32.320138 2185 raft_consensus.cc:348] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.320904 2185 raft_consensus.cc:374] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.320515 2324 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.321267 2185 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.321523 2324 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.322384 2185 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.323666 2324 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:32.324997 2185 ts_tablet_manager.cc:1428] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.047s user 0.009s sys 0.022s
I20260924 00:21:32.325217 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:32.325956 2185 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.326100 1992 raft_consensus.cc:2379] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:32.325357 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:32.320149 2188 tablet_bootstrap.cc:654] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.330442 2109 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.332212 2185 tablet_bootstrap.cc:654] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.333499 2324 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.344215 2441 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.344797 2441 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.346884 2441 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.350363 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:32.351022 1994 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.352167 2036 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:32.352074 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.356879 2441 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:32.352943 2139 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.357568 2441 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:32.357618 2141 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260924 00:21:32.358294 2141 pending_rounds.cc:76] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Aborting all ops after (but not including) 0
I20260924 00:21:32.358250 2441 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.358744 2141 pending_rounds.cc:97] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260924 00:21:32.359301 2141 raft_consensus.cc:2866] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:21:32.360946 2384 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:32.367511 2324 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.361227 2383 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.373775 2273 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:32.393812 2186 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.394301 2186 ts_tablet_manager.cc:1397] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.151s user 0.012s sys 0.013s
I20260924 00:21:32.398828 2186 raft_consensus.cc:348] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.400098 2186 raft_consensus.cc:374] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.400334 2186 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.401345 2186 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.409438 2186 ts_tablet_manager.cc:1428] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260924 00:21:32.410405 2186 tablet_bootstrap.cc:492] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.431500 2383 consensus_queue.cc:1035] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:32.424958 2188 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:32.436949 2188 ts_tablet_manager.cc:1397] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.241s user 0.013s sys 0.010s
I20260924 00:21:32.449640 2441 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.452212 2186 tablet_bootstrap.cc:654] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.463927 2188 raft_consensus.cc:348] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.466180 2188 raft_consensus.cc:374] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.466516 2188 raft_consensus.cc:729] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.467401 2188 consensus_queue.cc:260] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.469883 2188 ts_tablet_manager.cc:1428] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.032s user 0.009s sys 0.002s
I20260924 00:21:32.463949 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:32.470361 2145 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.471079 2188 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:32.476241 2384 consensus_queue.cc:1035] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:32.482254 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:32.482934 1993 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.490020 1993 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:32.491235 2188 tablet_bootstrap.cc:654] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.492952 2269 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:32.494516 2036 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:32.497524 2261 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:32.498533 2261 raft_consensus.cc:686] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:32.499367 2261 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.479851 2145 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:32.502058 2441 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.507359 2185 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.508023 2185 ts_tablet_manager.cc:1397] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.182s user 0.005s sys 0.018s
I20260924 00:21:32.512885 2211 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.513064 2185 raft_consensus.cc:348] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.513484 2211 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.516038 2185 raft_consensus.cc:374] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.517510 2185 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.518308 2185 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.519804 2185 ts_tablet_manager.cc:1428] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260924 00:21:32.520684 2185 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.528753 2211 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.531159 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:32.532976 1960 leader_election.cc:341] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.533327 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:32.537217 1960 leader_election.cc:341] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.537592 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.538832 2281 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.540060 2185 tablet_bootstrap.cc:654] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.542088 2186 tablet_bootstrap.cc:492] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.542562 2186 ts_tablet_manager.cc:1397] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.132s user 0.017s sys 0.014s
I20260924 00:21:32.545603 2186 raft_consensus.cc:348] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.546525 2186 raft_consensus.cc:374] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.546867 2186 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.547577 2186 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.554111 1891 catalog_manager.cc:5495] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:32.557296 2281 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.557818 2281 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.560178 2281 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.560763 2188 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:32.561218 2188 ts_tablet_manager.cc:1397] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.006s
I20260924 00:21:32.561820 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:32.562009 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:32.563226 1960 leader_election.cc:341] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:32.563835 1960 leader_election.cc:341] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.563683 2188 raft_consensus.cc:348] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.564262 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.559188 2186 ts_tablet_manager.cc:1428] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260924 00:21:32.565373 2186 tablet_bootstrap.cc:492] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.566668 2281 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.570044 2185 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.570513 2185 ts_tablet_manager.cc:1397] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.019s
I20260924 00:21:32.573596 2185 raft_consensus.cc:348] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.574309 2185 raft_consensus.cc:374] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.564401 2188 raft_consensus.cc:374] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.574573 2185 raft_consensus.cc:729] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.574707 2188 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.575517 2185 consensus_queue.cc:260] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.575320 2188 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.578423 2188 ts_tablet_manager.cc:1428] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260924 00:21:32.580395 2188 tablet_bootstrap.cc:492] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:32.583689 2185 ts_tablet_manager.cc:1428] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260924 00:21:32.586915 2186 tablet_bootstrap.cc:654] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.588742 2188 tablet_bootstrap.cc:654] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.601239 2186 tablet_bootstrap.cc:492] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.602077 2185 tablet_bootstrap.cc:492] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.602639 2186 ts_tablet_manager.cc:1397] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.002s
I20260924 00:21:32.606331 2186 raft_consensus.cc:348] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.607574 2186 raft_consensus.cc:374] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.608019 2186 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.608803 2186 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.610525 2202 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.611009 2202 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.613072 2202 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.615180 2188 tablet_bootstrap.cc:492] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:32.615686 2188 ts_tablet_manager.cc:1397] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.006s
I20260924 00:21:32.614964 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.616170 2145 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.616751 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:32.617542 1994 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.618973 2036 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:32.620365 2185 tablet_bootstrap.cc:654] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.620891 2186 ts_tablet_manager.cc:1428] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260924 00:21:32.620937 2383 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:32.621624 2383 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:32.621973 2383 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.622040 2186 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.627161 2188 raft_consensus.cc:348] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.627981 2188 raft_consensus.cc:374] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.628257 2188 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.628893 2188 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.629930 2186 tablet_bootstrap.cc:654] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.634198 2188 ts_tablet_manager.cc:1428] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.018s user 0.010s sys 0.001s
I20260924 00:21:32.632802 2383 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.635545 2188 tablet_bootstrap.cc:492] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:32.639365 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:32.640168 1994 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.640690 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:32.641363 2145 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.649204 2185 tablet_bootstrap.cc:492] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.649708 2185 ts_tablet_manager.cc:1397] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.000s
I20260924 00:21:32.650018 1994 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:32.651451 2036 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:32.652576 2202 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:32.653062 2202 raft_consensus.cc:686] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:32.653955 2384 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:32.653880 2202 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.655182 2188 tablet_bootstrap.cc:654] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.652237 2185 raft_consensus.cc:348] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.656441 2185 raft_consensus.cc:374] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.656759 2185 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.657356 2185 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.659876 2185 ts_tablet_manager.cc:1428] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260924 00:21:32.660735 2185 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.664999 2145 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:32.668576 2185 tablet_bootstrap.cc:654] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.665935 2383 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.674197 2186 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.674934 2186 ts_tablet_manager.cc:1397] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.002s
I20260924 00:21:32.675432 2185 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.675835 2185 ts_tablet_manager.cc:1397] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260924 00:21:32.681794 2186 raft_consensus.cc:348] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.682930 2186 raft_consensus.cc:374] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.683288 2186 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.682670 1892 catalog_manager.cc:5495] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:32.684139 2186 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.690073 2185 raft_consensus.cc:348] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.691051 2185 raft_consensus.cc:374] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.691320 2185 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.693935 2441 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:32.692157 2185 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.698755 2185 ts_tablet_manager.cc:1428] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.023s user 0.006s sys 0.003s
I20260924 00:21:32.699698 2185 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.707182 2185 tablet_bootstrap.cc:654] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.719523 2188 tablet_bootstrap.cc:492] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:32.719678 2186 ts_tablet_manager.cc:1428] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.044s user 0.005s sys 0.007s
I20260924 00:21:32.720017 2188 ts_tablet_manager.cc:1397] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.031s
I20260924 00:21:32.720639 2186 tablet_bootstrap.cc:492] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.722424 2188 raft_consensus.cc:348] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.723189 2188 raft_consensus.cc:374] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.723454 2188 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.724181 2188 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.725677 2269 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.726688 2269 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.732529 2188 ts_tablet_manager.cc:1428] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260924 00:21:32.733704 2188 tablet_bootstrap.cc:492] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:32.738770 2185 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.739253 2185 ts_tablet_manager.cc:1397] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.002s
I20260924 00:21:32.741199 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:32.741959 1994 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:32.742015 2269 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:32.743705 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:32.744108 2110 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:32.744575 2068 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:32.745096 2269 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:32.745558 2269 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:32.745939 2269 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.746078 2185 raft_consensus.cc:348] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.749482 2186 tablet_bootstrap.cc:654] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.749666 2185 raft_consensus.cc:374] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.750051 2185 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.750788 2185 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.759150 2185 ts_tablet_manager.cc:1428] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.020s user 0.008s sys 0.002s
I20260924 00:21:32.762626 2188 tablet_bootstrap.cc:654] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.769531 2185 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.793548 2185 tablet_bootstrap.cc:654] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.790300 2269 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.802062 2186 tablet_bootstrap.cc:492] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.803174 2186 ts_tablet_manager.cc:1397] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.004s
I20260924 00:21:32.804530 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:32.805290 1993 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.805907 2269 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:32.808013 2186 raft_consensus.cc:348] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.809088 2186 raft_consensus.cc:374] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.809448 2186 raft_consensus.cc:729] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.809652 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:32.810524 2067 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.810227 2186 consensus_queue.cc:260] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.813177 2186 ts_tablet_manager.cc:1428] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260924 00:21:32.814630 2186 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.821593 2185 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.822094 2185 ts_tablet_manager.cc:1397] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.007s
I20260924 00:21:32.822773 2188 tablet_bootstrap.cc:492] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:32.823211 2188 ts_tablet_manager.cc:1397] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.000s
I20260924 00:21:32.825459 2281 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.826298 2281 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.826963 2188 raft_consensus.cc:348] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.827839 2188 raft_consensus.cc:374] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.828145 2188 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.828739 2281 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.829660 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:32.832216 1960 leader_election.cc:341] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.832216 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.833323 2139 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:32.834124 1993 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:32.835670 2110 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:32.835904 2067 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:32.836722 2281 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.837303 2281 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.838800 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.839790 2388 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:32.840127 2388 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:32.840317 2281 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.840451 2388 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.841818 2186 tablet_bootstrap.cc:654] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.842590 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.843022 2269 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:32.843580 2269 raft_consensus.cc:686] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:21:32.844636 1960 leader_election.cc:341] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.845530 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:32.846772 1960 leader_election.cc:341] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.847179 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.838836 2185 raft_consensus.cc:348] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.847996 2185 raft_consensus.cc:374] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.848234 2281 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.848317 2185 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.838548 2188 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.848129 2388 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.849195 2185 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.852352 2185 ts_tablet_manager.cc:1428] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20260924 00:21:32.852306 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:32.846340 2269 consensus_queue.cc:237] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:32.854159 1960 leader_election.cc:341] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.855341 2388 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.856755 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:32.857461 2188 ts_tablet_manager.cc:1428] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.034s user 0.007s sys 0.001s
I20260924 00:21:32.853713 2185 tablet_bootstrap.cc:492] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.857739 2139 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.862649 2188 tablet_bootstrap.cc:492] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:32.871104 2139 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:32.872320 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.876253 2186 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.876739 2186 ts_tablet_manager.cc:1397] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.019s
I20260924 00:21:32.877031 2185 tablet_bootstrap.cc:654] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.880546 2188 tablet_bootstrap.cc:654] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.881212 2441 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.881677 2441 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.882618 2186 raft_consensus.cc:348] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.883502 2186 raft_consensus.cc:374] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.884672 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:32.885361 1993 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.884838 2186 raft_consensus.cc:729] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.886646 2036 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:32.885892 2281 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:32.887537 2281 raft_consensus.cc:686] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:32.890225 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.889775 2281 consensus_queue.cc:237] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.891016 2139 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.888173 2384 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:32.887655 2186 consensus_queue.cc:260] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.892917 2384 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:32.893392 2384 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.898662 2186 ts_tablet_manager.cc:1428] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.022s user 0.007s sys 0.005s
I20260924 00:21:32.900700 2186 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.908329 2388 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.908488 2186 tablet_bootstrap.cc:654] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.909202 2441 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.908861 2388 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.910888 2388 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.910337 1892 catalog_manager.cc:5495] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:32.912242 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:32.913991 1960 leader_election.cc:341] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.914935 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:32.916421 1960 leader_election.cc:341] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.916858 1960 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.917626 2388 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.916875 2384 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.932313 2185 tablet_bootstrap.cc:492] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:32.932804 2185 ts_tablet_manager.cc:1397] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.016s
I20260924 00:21:32.934559 2384 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.934396 1891 catalog_manager.cc:5495] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:32.935458 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:32.936087 1993 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.936241 2185 raft_consensus.cc:348] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.936919 2185 raft_consensus.cc:374] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.937206 2185 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.937835 2185 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.938534 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:32.939147 2145 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.939429 2388 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.940001 2185 ts_tablet_manager.cc:1428] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20260924 00:21:32.939862 2388 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.940855 2185 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:32.941746 2388 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.943321 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:32.943329 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:32.945192 1960 leader_election.cc:341] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:32.945999 1960 leader_election.cc:341] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:32.946576 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:32.947882 2388 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:32.952554 2441 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.953035 2441 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.955629 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:32.956288 1991 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.957818 2036 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:32.959533 2202 raft_consensus.cc:2781] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:32.959874 2202 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:32.960166 2202 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.966112 2441 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.970034 2186 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:32.970157 2202 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.972056 2185 tablet_bootstrap.cc:654] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.972285 2186 ts_tablet_manager.cc:1397] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.070s user 0.019s sys 0.011s
I20260924 00:21:32.974215 2188 tablet_bootstrap.cc:492] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:32.974916 2186 raft_consensus.cc:348] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.975698 2186 raft_consensus.cc:374] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.975963 2186 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.976243 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:32.976934 1991 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:32.976588 2186 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.977964 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.978650 2139 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:32.980563 2188 ts_tablet_manager.cc:1397] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.118s user 0.001s sys 0.018s
I20260924 00:21:32.986339 2188 raft_consensus.cc:348] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.987475 2188 raft_consensus.cc:374] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:32.987937 2188 raft_consensus.cc:729] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:32.988449 2385 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:32.989030 2186 ts_tablet_manager.cc:1428] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260924 00:21:32.988631 2188 consensus_queue.cc:260] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.988988 2385 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:32.990108 2186 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:32.991385 2188 ts_tablet_manager.cc:1428] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 00:21:32.991451 2385 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:32.992545 2188 tablet_bootstrap.cc:492] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:32.996346 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:32.997084 2186 tablet_bootstrap.cc:654] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.998974 2188 tablet_bootstrap.cc:654] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:32.999720 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.000373 1994 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:33.002521 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:33.003603 2261 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.004055 2261 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.004527 2261 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.006883 1991 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
W20260924 00:21:33.007792 2035 leader_election.cc:341] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.011843 2036 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:33.012483 2145 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:33.012878 2186 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:33.012981 2388 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.013748 2186 ts_tablet_manager.cc:1397] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.008s
I20260924 00:21:33.013638 2388 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.013904 2202 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.015673 2388 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.017792 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:33.019091 1960 leader_election.cc:341] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.020167 1993 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:33.022228 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:33.023425 1960 leader_election.cc:341] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.023859 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.024127 2186 raft_consensus.cc:348] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.024407 2202 raft_consensus.cc:2781] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.024922 2188 tablet_bootstrap.cc:492] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:33.025499 2202 raft_consensus.cc:686] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:33.025817 2188 ts_tablet_manager.cc:1397] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.003s
I20260924 00:21:33.024953 2186 raft_consensus.cc:374] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.026767 2202 consensus_queue.cc:237] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.028918 2281 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.027274 2186 raft_consensus.cc:729] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.026522 2036 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:33.031121 2186 consensus_queue.cc:260] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.035984 2383 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.037460 2383 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:33.040366 2383 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.042217 2186 ts_tablet_manager.cc:1428] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20260924 00:21:33.038973 2188 raft_consensus.cc:348] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.043608 2188 raft_consensus.cc:374] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.043932 2188 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.044655 2188 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.041522 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:33.045272 2384 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.045745 2185 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:33.045825 2139 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.046382 2185 ts_tablet_manager.cc:1397] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.106s user 0.021s sys 0.026s
I20260924 00:21:33.049815 2185 raft_consensus.cc:348] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.050592 2185 raft_consensus.cc:374] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.050242 2261 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.053776 2261 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.053987 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:21:33.055336 2035 leader_election.cc:341] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.050900 2185 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.057960 2185 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.059551 2185 ts_tablet_manager.cc:1428] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260924 00:21:33.060482 2188 ts_tablet_manager.cc:1428] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20260924 00:21:33.061546 2188 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:33.062068 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:33.062508 2185 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:33.062849 1991 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.071189 2188 tablet_bootstrap.cc:654] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.073937 2384 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.074692 2185 tablet_bootstrap.cc:654] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.053987 2139 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:33.079835 2186 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:33.085896 2186 tablet_bootstrap.cc:654] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.106420 2261 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.107208 2261 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.109815 2261 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.110889 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:33.076079 1991 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
W20260924 00:21:33.112257 2035 leader_election.cc:341] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.119621 2186 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:33.120236 2186 ts_tablet_manager.cc:1397] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.003s
I20260924 00:21:33.123394 2186 raft_consensus.cc:348] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.124322 2186 raft_consensus.cc:374] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.124647 2186 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.124950 2388 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.125535 2388 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.127614 2388 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.126967 1891 catalog_manager.cc:5495] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:33.130113 2186 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.132570 2186 ts_tablet_manager.cc:1428] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260924 00:21:33.133494 2186 tablet_bootstrap.cc:492] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:33.136035 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.136974 1992 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:33.138023 2188 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:33.138530 2188 ts_tablet_manager.cc:1397] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.077s user 0.019s sys 0.021s
I20260924 00:21:33.141180 2188 raft_consensus.cc:348] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.142040 2188 raft_consensus.cc:374] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.142334 2188 raft_consensus.cc:729] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.144505 2186 tablet_bootstrap.cc:654] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.144722 2188 consensus_queue.cc:260] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.148209 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:33.149977 2384 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.150918 2384 raft_consensus.cc:686] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:33.152423 2036 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:33.154251 2202 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.154657 2202 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.154974 2202 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.156558 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.152812 2384 consensus_queue.cc:237] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.159590 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:33.160255 2139 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:33.162482 2202 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.163648 1960 leader_election.cc:341] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.164137 2188 ts_tablet_manager.cc:1428] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260924 00:21:33.165441 2202 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.166193 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:21:33.167955 2035 leader_election.cc:341] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.169829 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:33.170595 1991 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.176398 2188 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:33.181706 1960 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.185201 2269 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.185774 2269 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.187943 2269 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.189630 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.190330 1992 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:33.191601 2110 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:33.191738 2202 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.189442 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.192724 2068 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:33.192387 2202 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.194192 2388 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.197190 2388 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.197677 2388 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.200263 2383 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:33.196178 2185 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:33.206591 2185 ts_tablet_manager.cc:1397] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.018s
I20260924 00:21:33.207175 2383 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:33.192410 2269 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.209367 2269 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.209919 2269 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.216452 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.217726 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:33.218132 2186 tablet_bootstrap.cc:492] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:33.219105 2186 ts_tablet_manager.cc:1397] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.086s user 0.038s sys 0.019s
W20260924 00:21:33.219319 2035 leader_election.cc:341] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.220890 2202 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.222015 2186 raft_consensus.cc:348] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.222795 2186 raft_consensus.cc:374] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.223070 2186 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.223634 2186 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.225788 2186 ts_tablet_manager.cc:1428] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 00:21:33.226681 2186 tablet_bootstrap.cc:492] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:33.212818 1991 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:33.230054 2185 raft_consensus.cc:348] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.230866 2185 raft_consensus.cc:374] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.231163 2185 raft_consensus.cc:729] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.231824 2185 consensus_queue.cc:260] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.233093 2186 tablet_bootstrap.cc:654] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.204487 1892 catalog_manager.cc:5495] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:33.250546 2185 ts_tablet_manager.cc:1428] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.044s user 0.008s sys 0.000s
I20260924 00:21:33.251573 2185 tablet_bootstrap.cc:492] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:33.226047 2388 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.255138 2388 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:33.249707 2036 leader_election.cc:341] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.256054 2036 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:33.257292 2036 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:33.258049 2384 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.259176 2384 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.259701 2384 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:33.260066 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:33.259701 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:33.260684 2384 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.262249 1960 leader_election.cc:341] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.259382 1892 catalog_manager.cc:5495] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:33.260712 2140 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.244441 2269 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.270269 2188 tablet_bootstrap.cc:654] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.277866 2186 tablet_bootstrap.cc:492] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:33.278354 2186 ts_tablet_manager.cc:1397] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.052s user 0.018s sys 0.031s
I20260924 00:21:33.278927 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:33.279091 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:33.279556 1992 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.279666 2068 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.280786 2186 raft_consensus.cc:348] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.281651 2186 raft_consensus.cc:374] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.281920 2186 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.282567 2186 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.289847 2388 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.290483 2388 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.290906 2211 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.283885 2269 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.293347 2388 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.294936 2068 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:33.296195 2109 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:33.306856 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:33.320492 1960 leader_election.cc:341] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.327673 2186 ts_tablet_manager.cc:1428] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260924 00:21:33.329048 2186 tablet_bootstrap.cc:492] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:33.326278 2434 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.337410 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.348775 2185 tablet_bootstrap.cc:654] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.357683 1892 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:33.355607 2269 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.361575 2434 raft_consensus.cc:686] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:33.364776 2269 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.365458 2434 consensus_queue.cc:237] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.367861 1960 leader_election.cc:341] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.369664 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.371358 2281 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.368973 2269 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.382818 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.386622 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:33.394121 2110 leader_election.cc:341] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.345651 1992 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
W20260924 00:21:33.405035 1960 consensus_peers.cc:487] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd -> Peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Couldn't send request to peer 948356bf66a941ef8249ecbd0ec94f6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260924 00:21:33.405853 2109 leader_election.cc:341] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.406315 2109 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.407351 2273 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.408371 2324 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.408921 2324 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.417301 2383 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.418927 2324 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.419055 2188 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:33.419643 2188 ts_tablet_manager.cc:1397] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.244s user 0.015s sys 0.006s
I20260924 00:21:33.417029 2186 tablet_bootstrap.cc:654] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.424623 2188 raft_consensus.cc:348] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.444845 2188 raft_consensus.cc:374] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.445277 2188 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.445797 2188 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.451223 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:21:33.437956 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.439522 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.463824 2269 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.464627 2273 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260924 00:21:33.466162 2269 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.468698 2269 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.465464 1993 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:33.449581 2140 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:33.474988 2269 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:33.475710 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.484272 2188 ts_tablet_manager.cc:1428] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.064s user 0.005s sys 0.003s
I20260924 00:21:33.483237 2211 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.487569 2211 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.485623 1894 catalog_manager.cc:5495] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:33.493237 1960 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.493881 2188 tablet_bootstrap.cc:492] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
W20260924 00:21:33.493283 2109 leader_election.cc:341] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.495935 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.496570 1993 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:33.497676 2211 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.497833 2281 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.498287 2281 raft_consensus.cc:686] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:33.499233 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.499558 2281 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.500880 1960 leader_election.cc:341] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.501011 2110 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.502105 2110 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.505666 2504 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.508173 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:33.524008 2202 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.509532 2434 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.535881 2185 tablet_bootstrap.cc:492] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:33.536355 2185 ts_tablet_manager.cc:1397] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.285s user 0.013s sys 0.022s
I20260924 00:21:33.538816 2273 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.529709 2140 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:33.539405 2434 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.539280 2273 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.539809 2434 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.549942 2273 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.553386 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.557541 2388 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.557679 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.557960 2388 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.558316 2388 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.561808 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.563372 1991 raft_consensus.cc:2379] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:33.548573 2211 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.550563 2504 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.570127 2211 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.580691 2185 raft_consensus.cc:348] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.582752 2185 raft_consensus.cc:374] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.583123 2185 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.589700 2110 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.583942 2185 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.605304 2188 tablet_bootstrap.cc:654] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.602312 2269 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.593783 2388 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.608879 2388 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.609771 2211 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.613298 2441 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.613849 2441 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.616680 2441 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.621728 2434 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.627161 2434 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.628072 2269 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.630777 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:33.635315 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:21:33.636489 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 62820.8us
I20260924 00:21:33.639252 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.639941 2067 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:33.641538 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:33.639400 2035 leader_election.cc:341] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.642130 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:33.642930 2140 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:33.644536 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:33.647557 2383 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.648562 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.649322 1993 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:33.655575 2186 tablet_bootstrap.cc:492] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:33.656075 2186 ts_tablet_manager.cc:1397] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.327s user 0.011s sys 0.007s
I20260924 00:21:33.659984 2186 raft_consensus.cc:348] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.660714 2186 raft_consensus.cc:374] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.660995 2186 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.661664 2186 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.664532 1960 leader_election.cc:341] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.666042 1960 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:33.666792 2388 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260924 00:21:33.667120 1960 leader_election.cc:341] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.667212 2388 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.667580 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.667814 2388 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.668494 2513 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.644627 1891 catalog_manager.cc:5495] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:33.670866 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:33.671707 1992 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:33.652321 2188 tablet_bootstrap.cc:492] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:33.673050 2188 ts_tablet_manager.cc:1397] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.179s user 0.007s sys 0.009s
I20260924 00:21:33.675559 2188 raft_consensus.cc:348] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.676021 2188 raft_consensus.cc:374] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.676216 2188 raft_consensus.cc:729] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.676767 2188 consensus_queue.cc:260] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.678140 2188 ts_tablet_manager.cc:1428] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 00:21:33.679388 2186 ts_tablet_manager.cc:1428] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20260924 00:21:33.684202 2388 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.687672 2109 leader_election.cc:341] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.689934 2109 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.691149 2185 ts_tablet_manager.cc:1428] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.154s user 0.005s sys 0.007s
I20260924 00:21:33.691788 2388 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.691730 2202 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.692764 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:33.694506 2385 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:21:33.694922 1960 leader_election.cc:341] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.692935 2273 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.696293 2273 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.697877 2385 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.700053 2273 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.698189 2185 tablet_bootstrap.cc:492] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:33.697964 2036 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:33.702376 2385 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.698967 2186 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:33.702785 2385 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.704442 2385 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.706077 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:33.707423 1993 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:33.708971 2110 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:33.703053 2273 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.710047 2273 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:33.710376 2273 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:33.710789 2273 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.718353 2273 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.725354 2188 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:33.726825 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:33.727465 1992 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.729568 2385 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.731469 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:33.732101 2068 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.736632 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:33.737231 1994 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:33.740069 2273 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:33.749087 2388 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:33.753425 1994 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:33.733542 2385 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.767688 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:33.754665 2388 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.758894 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:33.783635 2036 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:21:33.790117 2109 leader_election.cc:341] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:33.792873 2035 leader_election.cc:341] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.790377 2384 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.797783 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:33.806738 1960 leader_election.cc:341] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.815073 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:21:33.817118 2109 leader_election.cc:341] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.795696 2388 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:33.805603 2384 raft_consensus.cc:686] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:33.830009 2188 tablet_bootstrap.cc:654] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.834120 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:21:33.836320 1960 leader_election.cc:341] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:33.836771 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.838868 2515 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:33.843269 2186 tablet_bootstrap.cc:654] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.831661 2384 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.845521 1992 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:33.847100 2110 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:33.860028 2185 tablet_bootstrap.cc:654] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:33.829535 2068 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:33.859910 2434 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.862792 2434 raft_consensus.cc:686] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:33.863173 1960 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:33.863628 2434 consensus_queue.cc:237] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.863960 2388 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:33.864694 2388 raft_consensus.cc:686] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:33.866055 2388 consensus_queue.cc:237] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:33.868005 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 73134.5us
I20260924 00:21:33.896343 2186 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:33.896879 2186 ts_tablet_manager.cc:1397] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.217s user 0.011s sys 0.006s
I20260924 00:21:33.899816 2261 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:33.904120 2186 raft_consensus.cc:348] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.905759 1891 catalog_manager.cc:5495] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:33.906832 2186 raft_consensus.cc:374] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.908927 2186 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.914629 1892 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:33.909772 2186 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.922585 2523 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:33.950385 2186 ts_tablet_manager.cc:1428] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.053s user 0.006s sys 0.001s
I20260924 00:21:33.959474 2186 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:33.961764 2188 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:33.962402 2188 ts_tablet_manager.cc:1397] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.237s user 0.021s sys 0.009s
I20260924 00:21:33.971071 2188 raft_consensus.cc:348] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.972273 2188 raft_consensus.cc:374] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:33.972582 2188 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:33.974215 2188 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:33.989810 2324 consensus_queue.cc:1035] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:33.991281 2324 consensus_queue.cc:1035] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:33.988306 1892 catalog_manager.cc:5495] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:34.004299 2188 ts_tablet_manager.cc:1428] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.042s user 0.005s sys 0.005s
I20260924 00:21:34.005617 2188 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.007261 2186 tablet_bootstrap.cc:654] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.014492 2185 tablet_bootstrap.cc:492] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:34.016948 2188 tablet_bootstrap.cc:654] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.021395 2185 ts_tablet_manager.cc:1397] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.323s user 0.022s sys 0.038s
I20260924 00:21:34.026360 2185 raft_consensus.cc:348] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.027153 2185 raft_consensus.cc:374] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.027429 2185 raft_consensus.cc:729] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.028131 2185 consensus_queue.cc:260] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.030841 2185 ts_tablet_manager.cc:1428] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260924 00:21:34.039798 2186 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.040354 2186 ts_tablet_manager.cc:1397] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.005s
I20260924 00:21:34.041448 2185 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:34.042927 2186 raft_consensus.cc:348] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.043684 2186 raft_consensus.cc:374] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.043962 2186 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.045200 2186 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.047240 2188 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.048069 2188 ts_tablet_manager.cc:1397] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.001s
I20260924 00:21:34.048154 2185 tablet_bootstrap.cc:654] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.049602 2186 ts_tablet_manager.cc:1428] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 00:21:34.050500 2186 tablet_bootstrap.cc:492] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.050943 2188 raft_consensus.cc:348] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.051863 2188 raft_consensus.cc:374] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.052251 2188 raft_consensus.cc:729] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.054905 2188 consensus_queue.cc:260] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.059317 2188 ts_tablet_manager.cc:1428] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260924 00:21:34.060948 2188 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.065491 2186 tablet_bootstrap.cc:654] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
W20260924 00:21:34.078821 1960 consensus_peers.cc:487] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd -> Peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Couldn't send request to peer 948356bf66a941ef8249ecbd0ec94f6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:34.084887 2188 tablet_bootstrap.cc:654] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.086683 2185 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:34.088794 2185 ts_tablet_manager.cc:1397] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.048s user 0.024s sys 0.012s
I20260924 00:21:34.089529 2388 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.090008 2388 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.092751 2185 raft_consensus.cc:348] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.094687 2185 raft_consensus.cc:374] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.095043 2185 raft_consensus.cc:729] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.095234 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.096365 2185 consensus_queue.cc:260] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:34.097302 1960 leader_election.cc:341] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.098019 2388 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.099684 2185 ts_tablet_manager.cc:1428] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.010s user 0.002s sys 0.005s
I20260924 00:21:34.099452 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.101256 2185 tablet_bootstrap.cc:492] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:34.102958 2388 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.103586 2388 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.110630 2388 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.114588 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.112957 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:21:34.116690 1960 leader_election.cc:341] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.116731 2141 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:34.120128 1960 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.121374 2388 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.121666 2388 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.121856 2388 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.126089 2185 tablet_bootstrap.cc:654] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.108579 2065 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:34.131284 2388 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.132798 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:34.133919 2281 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.134320 2281 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.134812 2281 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.132316 2188 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.135998 2188 ts_tablet_manager.cc:1397] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.075s user 0.024s sys 0.027s
I20260924 00:21:34.136237 2185 tablet_bootstrap.cc:492] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:34.136050 2515 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:34.136723 2185 ts_tablet_manager.cc:1397] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.001s
I20260924 00:21:34.138700 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:34.139945 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:34.140580 2143 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:34.145788 1960 leader_election.cc:341] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.148232 2188 raft_consensus.cc:348] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.149353 2188 raft_consensus.cc:374] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.149748 2188 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.149892 2143 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:34.151365 1960 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.153823 2515 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.154371 2515 raft_consensus.cc:686] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:34.153537 2185 raft_consensus.cc:348] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.155603 2185 raft_consensus.cc:374] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.155210 2515 consensus_queue.cc:237] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.155947 2185 raft_consensus.cc:729] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.157720 2188 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.160682 2188 ts_tablet_manager.cc:1428] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20260924 00:21:34.157018 2185 consensus_queue.cc:260] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.164023 2185 ts_tablet_manager.cc:1428] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.027s user 0.008s sys 0.001s
I20260924 00:21:34.164916 2388 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.165364 2185 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:34.174516 2185 tablet_bootstrap.cc:654] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.166316 2188 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.167150 2186 tablet_bootstrap.cc:492] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.176790 2186 ts_tablet_manager.cc:1397] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.126s user 0.012s sys 0.006s
I20260924 00:21:34.192332 2186 raft_consensus.cc:348] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.193511 2186 raft_consensus.cc:374] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.167006 2281 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.200330 2281 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.193838 2186 raft_consensus.cc:729] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.201709 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:34.207515 2065 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.208156 2186 consensus_queue.cc:260] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.203706 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:34.202425 2185 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:34.223721 2185 ts_tablet_manager.cc:1397] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.002s
I20260924 00:21:34.224637 2186 ts_tablet_manager.cc:1428] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260924 00:21:34.225263 2514 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.225651 2186 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.225713 2514 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.226716 2185 raft_consensus.cc:348] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.217002 2504 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.232710 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.227743 2514 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.231890 1893 catalog_manager.cc:5495] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:34.232574 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.233414 2504 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.234845 2145 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
W20260924 00:21:34.234866 1960 leader_election.cc:341] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.238893 2504 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:34.207528 2188 tablet_bootstrap.cc:654] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.217127 2065 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:34.240051 2185 raft_consensus.cc:374] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.240338 2185 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
W20260924 00:21:34.241266 1960 leader_election.cc:341] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.242012 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:34.240932 2185 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.244108 2514 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.244864 2514 raft_consensus.cc:686] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:34.245692 2514 consensus_queue.cc:237] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.247268 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.248220 1992 raft_consensus.cc:2379] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 1: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:21:34.253078 2185 ts_tablet_manager.cc:1428] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260924 00:21:34.254045 2185 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: Bootstrap starting.
I20260924 00:21:34.255231 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.266481 2185 tablet_bootstrap.cc:654] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.233345 2186 tablet_bootstrap.cc:654] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.272354 2188 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.272831 2188 ts_tablet_manager.cc:1397] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.107s user 0.013s sys 0.001s
I20260924 00:21:34.274718 2188 raft_consensus.cc:348] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.275326 2188 raft_consensus.cc:374] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.275559 2188 raft_consensus.cc:729] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.276132 2188 consensus_queue.cc:260] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.278098 2188 ts_tablet_manager.cc:1428] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260924 00:21:34.281216 2188 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.282160 2145 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.282399 1960 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.283828 2514 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.284230 2514 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.285036 2514 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.288894 2188 tablet_bootstrap.cc:654] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.291925 2514 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.294734 2186 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.295255 2186 ts_tablet_manager.cc:1397] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.008s
I20260924 00:21:34.298938 2514 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.306504 2523 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:34.307201 2109 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.307832 2188 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.308476 2504 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:34.308730 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:34.309671 2140 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.310544 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:34.310773 2186 raft_consensus.cc:348] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.311537 2186 raft_consensus.cc:374] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.311818 2186 raft_consensus.cc:729] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.312034 2384 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:34.312474 2186 consensus_queue.cc:260] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:34.316046 1960 leader_election.cc:341] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.316314 2140 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:34.319299 2185 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: No bootstrap required, opened a new log
I20260924 00:21:34.319892 2185 ts_tablet_manager.cc:1397] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.010s
I20260924 00:21:34.319324 1960 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.321224 1892 catalog_manager.cc:5495] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:34.322252 2388 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.322443 2186 ts_tablet_manager.cc:1428] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.027s user 0.009s sys 0.000s
I20260924 00:21:34.322731 2388 raft_consensus.cc:686] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:34.323397 2186 tablet_bootstrap.cc:492] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.323422 2388 consensus_queue.cc:237] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.308261 2188 ts_tablet_manager.cc:1397] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20260924 00:21:34.330391 2186 tablet_bootstrap.cc:654] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.331153 2188 raft_consensus.cc:348] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.331861 2188 raft_consensus.cc:374] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.332118 2188 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.332716 2188 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.334805 2188 ts_tablet_manager.cc:1428] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 00:21:34.335650 2188 tablet_bootstrap.cc:492] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.328686 2185 raft_consensus.cc:348] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.339037 2185 raft_consensus.cc:374] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.339287 2185 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.339892 2185 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.342465 2185 ts_tablet_manager.cc:1428] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: Time spent starting tablet: real 0.022s user 0.004s sys 0.001s
I20260924 00:21:34.345480 2509 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.345995 2509 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.348809 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.349596 1992 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:34.350754 2110 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:34.351545 2434 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.351873 2434 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.352191 2434 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.353008 2509 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:21:34.353484 2109 consensus_peers.cc:487] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 -> Peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Couldn't send request to peer 948356bf66a941ef8249ecbd0ec94f6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:34.359673 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.360706 2063 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:34.364933 2434 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.373697 2384 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.374583 2384 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.376542 2188 tablet_bootstrap.cc:654] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.379036 2261 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.379528 2261 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.380782 2434 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:34.381430 2261 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.382845 2384 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.385484 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.385497 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:34.386974 1991 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.388579 2388 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.388190 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:34.389034 2388 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.390849 2388 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.389346 2063 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.397380 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.398348 1993 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:34.400130 1991 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:34.402897 2110 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:34.404269 2434 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.404757 2434 raft_consensus.cc:686] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:34.405597 2434 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.384982 1892 catalog_manager.cc:5495] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:34.409742 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.411002 2140 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:34.409754 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.414253 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.415398 2140 raft_consensus.cc:2379] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:34.416754 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.423808 2063 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:34.426554 2035 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:34.435863 2186 tablet_bootstrap.cc:492] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.436372 2186 ts_tablet_manager.cc:1397] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.113s user 0.059s sys 0.033s
I20260924 00:21:34.436257 2384 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.436654 2035 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:34.437655 2261 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.439037 2186 raft_consensus.cc:348] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.439769 2186 raft_consensus.cc:374] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.440040 2186 raft_consensus.cc:729] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.440641 2186 consensus_queue.cc:260] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:34.429961 1960 leader_election.cc:341] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
W20260924 00:21:34.442299 1960 consensus_peers.cc:487] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd -> Peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Couldn't send request to peer 8c129c01e1c047628f6a87b001aa7258. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260924 00:21:34.446698 1960 leader_election.cc:341] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.447156 1960 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.442857 2186 ts_tablet_manager.cc:1428] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 00:21:34.430994 2188 tablet_bootstrap.cc:492] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.448014 2188 ts_tablet_manager.cc:1397] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.003s
I20260924 00:21:34.448351 2186 tablet_bootstrap.cc:492] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.448519 2261 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:34.454694 2188 raft_consensus.cc:348] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.455497 2188 raft_consensus.cc:374] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.455796 2188 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.455353 1892 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:34.457544 2388 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:34.459436 2384 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:34.459851 2188 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.460929 2186 tablet_bootstrap.cc:654] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.462409 2188 ts_tablet_manager.cc:1428] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20260924 00:21:34.463369 2188 tablet_bootstrap.cc:492] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.474782 2188 tablet_bootstrap.cc:654] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.487880 2186 tablet_bootstrap.cc:492] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.488351 2186 ts_tablet_manager.cc:1397] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.040s user 0.021s sys 0.010s
I20260924 00:21:34.491701 2186 raft_consensus.cc:348] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.492488 2186 raft_consensus.cc:374] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.492763 2186 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.493454 2186 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.495272 2514 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:34.500430 2186 ts_tablet_manager.cc:1428] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260924 00:21:34.501557 2186 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.501820 2065 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.508101 2186 tablet_bootstrap.cc:654] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.509451 2509 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:34.508126 2188 tablet_bootstrap.cc:492] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.511369 2188 ts_tablet_manager.cc:1397] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.002s
I20260924 00:21:34.514112 2188 raft_consensus.cc:348] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.514855 2188 raft_consensus.cc:374] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.515161 2188 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.516556 2188 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.518241 2188 ts_tablet_manager.cc:1428] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:21:34.519168 2188 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.525731 2186 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.526292 2186 ts_tablet_manager.cc:1397] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s
I20260924 00:21:34.529317 2186 raft_consensus.cc:348] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.530084 2186 raft_consensus.cc:374] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.530367 2186 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.531019 2186 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.537508 2384 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.538262 2186 ts_tablet_manager.cc:1428] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260924 00:21:34.538051 2384 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.539244 2186 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.540589 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.541272 1993 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:34.542520 2036 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:34.542878 2384 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.543473 2384 raft_consensus.cc:2781] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.543792 2384 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.544324 2384 raft_consensus.cc:3037] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.545441 2202 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.545292 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.545881 2202 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.546144 2141 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:34.547731 2202 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.550433 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.552347 2186 tablet_bootstrap.cc:654] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.551690 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.557834 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:34.559729 2202 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:34.590372 2434 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.591005 2434 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.607154 2188 tablet_bootstrap.cc:654] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.611408 2261 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:34.611908 2261 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.619695 2384 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.622172 2261 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.622453 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.624619 2384 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.626329 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:34.627002 2145 raft_consensus.cc:3037] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.628638 2186 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.629127 2186 ts_tablet_manager.cc:1397] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.007s
W20260924 00:21:34.631608 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:21:34.631608 2186 raft_consensus.cc:348] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.634390 2145 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:34.636000 2434 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:34.634238 2186 raft_consensus.cc:374] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.636511 2186 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.637344 2186 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.639479 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.644634 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.647240 2186 ts_tablet_manager.cc:1428] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260924 00:21:34.647239 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:34.648123 1995 raft_consensus.cc:3037] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.648432 2186 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.648322 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:34.649240 2384 raft_consensus.cc:2781] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.649736 2384 raft_consensus.cc:686] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:34.650457 2384 consensus_queue.cc:237] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.655392 1995 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:34.658787 2186 tablet_bootstrap.cc:654] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.662567 2036 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:34.679049 2202 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:34.682732 2388 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:34.688409 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.696837 2186 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.697458 2186 ts_tablet_manager.cc:1397] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.000s
I20260924 00:21:34.697561 2434 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.698710 2434 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.700326 2186 raft_consensus.cc:348] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.701213 2186 raft_consensus.cc:374] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.701551 2186 raft_consensus.cc:729] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.701992 2434 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:34.701874 2063 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.702280 2186 consensus_queue.cc:260] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.704770 2186 ts_tablet_manager.cc:1428] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260924 00:21:34.705006 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.705828 1991 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:34.706593 2388 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.707108 2388 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.708905 2388 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.711591 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.712113 2109 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.712507 2110 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:34.712890 2186 tablet_bootstrap.cc:492] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258: Bootstrap starting.
I20260924 00:21:34.716296 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.716995 2141 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 0.
I20260924 00:21:34.718405 2434 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.718827 2434 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.719183 2434 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.719473 2388 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:34.720501 2186 tablet_bootstrap.cc:654] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.722667 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.728091 2434 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.731074 2324 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 00:21:34.735244 2109 leader_election.cc:341] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.736166 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:34.740140 2388 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.740547 2388 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.740957 2388 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.745072 2141 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:34.756461 1960 leader_election.cc:341] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.756745 2188 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.757622 2188 ts_tablet_manager.cc:1397] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.239s user 0.049s sys 0.028s
I20260924 00:21:34.760321 2188 raft_consensus.cc:348] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.761023 2188 raft_consensus.cc:374] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.761309 2188 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.761996 2188 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.764532 2188 ts_tablet_manager.cc:1428] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 00:21:34.766907 2281 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:34.768640 2188 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.771607 2281 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:34.780597 2188 tablet_bootstrap.cc:654] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.780896 1892 catalog_manager.cc:5495] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:34.783617 2388 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.787022 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:34.785380 2388 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.791332 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:34.792119 1995 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.792220 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:34.792896 2140 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.793112 2188 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.793591 2188 ts_tablet_manager.cc:1397] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.000s
I20260924 00:21:34.795923 2066 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.800938 2188 raft_consensus.cc:348] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.801831 2188 raft_consensus.cc:374] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.802132 2188 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
W20260924 00:21:34.797253 1960 leader_election.cc:341] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.802933 1995 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:34.805099 2140 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:34.807483 2188 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.811479 2188 ts_tablet_manager.cc:1428] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260924 00:21:34.812757 2188 tablet_bootstrap.cc:492] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.813962 2202 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.814641 2202 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.818235 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.818976 1995 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 0.
I20260924 00:21:34.821640 2202 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.823123 2036 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:34.825874 2560 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.826296 2560 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.826648 2560 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.828648 2110 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:34.829804 2324 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.830468 2324 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:34.831641 2324 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.834331 2188 tablet_bootstrap.cc:654] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.839181 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:34.840183 2388 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.840592 2388 raft_consensus.cc:686] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:34.841318 2388 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:34.846647 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 58852us
I20260924 00:21:34.852144 2434 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:21:34.875489 1960 consensus_peers.cc:487] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd -> Peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Couldn't send request to peer 948356bf66a941ef8249ecbd0ec94f6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:34.888592 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:34.898377 2281 consensus_queue.cc:1035] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:34.888286 2560 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.908103 2560 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.914242 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.917842 2186 tablet_bootstrap.cc:492] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258: No bootstrap required, opened a new log
I20260924 00:21:34.918398 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:34.919049 1991 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.918419 2186 ts_tablet_manager.cc:1397] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258: Time spent bootstrapping tablet: real 0.206s user 0.036s sys 0.078s
I20260924 00:21:34.923629 2186 raft_consensus.cc:348] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.924335 2186 raft_consensus.cc:374] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.924600 2186 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.925276 2186 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.927627 2186 ts_tablet_manager.cc:1428] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 00:21:34.932577 2188 tablet_bootstrap.cc:492] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.952543 2188 ts_tablet_manager.cc:1397] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.140s user 0.014s sys 0.010s
I20260924 00:21:34.939023 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:34.943286 2324 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.953293 2504 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:34.953521 2324 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.954090 2141 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.955397 2324 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:34.955159 2188 raft_consensus.cc:348] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.955821 2188 raft_consensus.cc:374] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:34.956107 2188 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:34.956676 2188 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.958218 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:34.958833 1993 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 0.
I20260924 00:21:34.958887 2188 ts_tablet_manager.cc:1428] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 00:21:34.959592 2139 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:34.945436 2035 leader_election.cc:341] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.962977 2188 tablet_bootstrap.cc:492] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:34.945447 1991 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
W20260924 00:21:34.948819 2109 leader_election.cc:341] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.960047 2110 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:34.943203 1892 catalog_manager.cc:5495] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:34.971354 2188 tablet_bootstrap.cc:654] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:34.974128 2560 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:34.964439 2036 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:34.964643 2324 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:21:34.964836 2509 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:34.977787 2202 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:34.978310 2202 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:34.977589 2560 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.979027 2202 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.980036 2560 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:34.981170 2324 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:34.981551 2324 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:34.981880 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:34.985045 2139 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:34.994447 2188 tablet_bootstrap.cc:492] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:34.994822 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:34.995555 2188 ts_tablet_manager.cc:1397] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.004s
I20260924 00:21:34.993914 2324 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:34.998927 2324 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:21:34.999365 2109 leader_election.cc:341] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:34.998358 2188 raft_consensus.cc:348] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.000968 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:35.001605 2188 raft_consensus.cc:374] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:35.002090 2188 raft_consensus.cc:729] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:35.002835 2188 consensus_queue.cc:260] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.004987 2188 ts_tablet_manager.cc:1428] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 00:21:35.005968 2188 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Bootstrap starting.
I20260924 00:21:35.054373 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:35.064710 1991 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:35.070032 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:35.071228 1992 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:21:35.059809 2109 leader_election.cc:341] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:21:35.098199 2036 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:35.110661 2202 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:35.121594 2580 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:35.124730 1992 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:35.122394 2580 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.127033 2580 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:35.129397 2202 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:35.130314 2110 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:35.132907 2585 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:21:35.136559 2188 tablet_bootstrap.cc:654] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Neither blocks nor log segments found. Creating new log.
I20260924 00:21:35.159109 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:35.144260 2585 raft_consensus.cc:686] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:35.150125 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:35.165441 2188 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: No bootstrap required, opened a new log
I20260924 00:21:35.164692 1890 catalog_manager.cc:5495] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:35.166105 2188 ts_tablet_manager.cc:1397] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent bootstrapping tablet: real 0.160s user 0.012s sys 0.007s
I20260924 00:21:35.170702 2188 raft_consensus.cc:348] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.171607 2188 raft_consensus.cc:374] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:21:35.171913 2188 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Initialized, Role: FOLLOWER
I20260924 00:21:35.173465 2188 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.178095 2188 ts_tablet_manager.cc:1428] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
I20260924 00:21:35.185901 2036 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:35.187004 2202 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:35.206257 2585 consensus_queue.cc:237] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.235774 2560 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:35.269328 2523 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:35.269899 2523 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.272267 2523 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:35.279799 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:35.280829 2143 raft_consensus.cc:2379] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:35.284986 2586 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:35.300845 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:35.288750 1890 catalog_manager.cc:5495] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:35.305750 1890 catalog_manager.cc:5495] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:35.314046 2514 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:35.317397 2560 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:35.333163 2036 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:35.334052 2384 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:35.342013 2504 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:35.342360 2384 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:35.357674 2560 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:35.358222 2560 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.362903 2560 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:35.363983 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:35.371766 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:35.372634 1991 raft_consensus.cc:2379] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 1: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:35.379448 2036 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:35.380573 2560 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:35.387972 2560 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:21:35.396817 2281 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:35.413964 2065 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:21:35.453261 2384 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:35.467056 2384 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:35.635474 2515 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:35.635927 2515 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.637909 2515 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:35.638653 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:35.639371 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:35.639343 2065 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:35.640038 2141 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:35.640908 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:35.641652 2515 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:35.641989 2515 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:35.642261 2515 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:35.655109 2515 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.656919 2515 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:35.657714 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:35.658319 2065 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:35.659191 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:35.659775 2141 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:35.667392 2065 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
W20260924 00:21:35.669235 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260924 00:21:35.670045 2141 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:35.670356 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:35.671159 2515 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:35.671674 2515 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:35.672500 2515 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:35.675025 2586 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:35.688614 1890 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 0 to 2, leader changed from <none> to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 2 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:35.689459 2509 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:36.114054 2063 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260924 00:21:36.115370 2515 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:21:36.126214 2141 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260924 00:21:36.127898 2515 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:21:36.498929 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:21:36.531629 2603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:21:36.536962 812 server_base.cc:1034] running on GCE node
W20260924 00:21:36.540208 2606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:21:36.540267 2604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:21:36.542752 812 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260924 00:21:36.543040 812 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260924 00:21:36.543233 812 hybrid_clock.cc:648] HybridClock initialized: now 1790209296543221 us; error 0 us; skew 500 ppm
I20260924 00:21:36.543776 812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:21:36.547601 812 webserver.cc:460] Webserver started at http://127.0.203.4:35145/ using document root <none> and password file <none>
I20260924 00:21:36.548228 812 fs_manager.cc:359] Metadata directory not provided
I20260924 00:21:36.548406 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:21:36.548663 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:21:36.550058 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-3-root/instance:
uuid: "9e79063532384ada8ea8f8205478adcf"
format_stamp: "Formatted at 2026-09-24 00:21:36 on dist-test-slave-r18p"
I20260924 00:21:36.555881 812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 00:21:36.559808 2611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:36.560699 812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 00:21:36.561061 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-3-root
uuid: "9e79063532384ada8ea8f8205478adcf"
format_stamp: "Formatted at 2026-09-24 00:21:36 on dist-test-slave-r18p"
I20260924 00:21:36.561432 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:21:36.576049 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:21:36.577768 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:21:36.579522 812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:21:36.582839 812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:21:36.583132 812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:36.583429 812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:21:36.583647 812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:21:36.678681 812 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.4:42667
I20260924 00:21:36.679028 2673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.4:42667 every 8 connection(s)
I20260924 00:21:36.748282 812 auto_leader_rebalancer-test.cc:413] add a tserver: 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:36.767248 2675 heartbeater.cc:347] Connected to a master server at 127.0.203.62:36337
I20260924 00:21:36.767736 2675 heartbeater.cc:464] Registering TS with master...
I20260924 00:21:36.768631 2675 heartbeater.cc:511] Master 127.0.203.62:36337 requested a full tablet report, sending...
I20260924 00:21:36.770830 1891 ts_manager.cc:194] Registered new tserver with Master: 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:36.772651 1891 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37650
I20260924 00:21:37.775785 2675 heartbeater.cc:503] Master 127.0.203.62:36337 was elected leader, sending a full tablet report...
I20260924 00:21:37.844213 1994 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:37.853065 2063 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:37.853574 2141 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:37.854851 2679 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:37.855801 2515 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:37.874564 2679 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:37.875939 2063 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:37.878489 2143 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:37.888556 1992 consensus_queue.cc:237] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:37.894274 1891 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260924 00:21:37.915432 1960 consensus_peers.cc:487] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 217f6c4b9f5c44d0b3856a8d131557cd. This is attempt 1: this message will repeat every 5th retry.
W20260924 00:21:37.960579 1960 consensus_peers.cc:487] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66641de0607c476a958313ea66dff4db. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:37.974262 2141 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:37.975912 2515 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:37.980083 2068 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:37.981881 2679 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:37.988629 2515 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:37.991886 2141 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.013916 2068 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.032648 1892 catalog_manager.cc:5495] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.034210 2141 consensus_queue.cc:237] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.043830 1992 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:38.045475 2693 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:38.060729 2068 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:38.063084 2693 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:38.064438 2694 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.066154 1992 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.085341 1994 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.089901 1892 catalog_manager.cc:5495] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
W20260924 00:21:38.092509 2109 consensus_peers.cc:487] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91be317234ca451fad76824aef0861a6. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:38.092561 2068 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.112039 2139 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:38.118382 2068 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:38.118856 2684 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260924 00:21:38.120592 1960 consensus_peers.cc:487] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cb231d6aeb14f4d83ab20c17ea2f9e1. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:38.122632 2515 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:38.132444 2515 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.134217 2066 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.133947 2141 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:38.148027 1890 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 2 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:38.307622 2708 ts_tablet_manager.cc:927] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:38.312793 2708 tablet_copy_client.cc:316] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
I20260924 00:21:38.332329 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet 217f6c4b9f5c44d0b3856a8d131557cd from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:21:38.332994 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet 217f6c4b9f5c44d0b3856a8d131557cd from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-217f6c4b9f5c44d0b3856a8d131557cd
I20260924 00:21:38.339779 2006 tablet_copy_source_session.cc:205] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:38.342970 2708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217f6c4b9f5c44d0b3856a8d131557cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:38.357448 2708 tablet_copy_client.cc:791] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:38.358181 2708 tablet_copy_client.cc:655] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:38.362973 2708 tablet_copy_client.cc:531] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:38.370494 2708 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:38.392416 2708 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:38.393349 2708 tablet_bootstrap.cc:492] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:38.393975 2708 ts_tablet_manager.cc:1397] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.008s
I20260924 00:21:38.396854 2708 raft_consensus.cc:348] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.397667 2708 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:38.398325 2708 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.401044 2708 ts_tablet_manager.cc:1428] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260924 00:21:38.405233 2006 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-217f6c4b9f5c44d0b3856a8d131557cd received from {username='slave'} at 127.0.0.1:42938
I20260924 00:21:38.405694 2006 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-217f6c4b9f5c44d0b3856a8d131557cd on tablet 217f6c4b9f5c44d0b3856a8d131557cd with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:38.572700 2708 ts_tablet_manager.cc:927] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:38.576450 2708 tablet_copy_client.cc:316] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
I20260924 00:21:38.578039 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet 66641de0607c476a958313ea66dff4db from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:21:38.578560 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet 66641de0607c476a958313ea66dff4db from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-66641de0607c476a958313ea66dff4db
I20260924 00:21:38.586064 2006 tablet_copy_source_session.cc:205] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:38.589331 2708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66641de0607c476a958313ea66dff4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:38.603816 2708 tablet_copy_client.cc:791] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:38.604434 2708 tablet_copy_client.cc:655] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:38.608341 2708 tablet_copy_client.cc:531] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:38.619344 2708 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:38.635380 2714 ts_tablet_manager.cc:927] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:38.640311 2708 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:38.641384 2708 tablet_bootstrap.cc:492] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:38.642207 2708 ts_tablet_manager.cc:1397] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.012s
I20260924 00:21:38.644860 2708 raft_consensus.cc:348] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.646441 2708 raft_consensus.cc:729] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:38.647342 2708 consensus_queue.cc:260] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.650110 2717 ts_tablet_manager.cc:927] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:38.648344 2714 tablet_copy_client.cc:316] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:21:38.658581 2717 tablet_copy_client.cc:316] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
I20260924 00:21:38.660271 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:21:38.659066 2708 ts_tablet_manager.cc:1428] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260924 00:21:38.660851 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-6cb231d6aeb14f4d83ab20c17ea2f9e1
I20260924 00:21:38.665884 2005 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-66641de0607c476a958313ea66dff4db received from {username='slave'} at 127.0.0.1:42938
I20260924 00:21:38.666283 2005 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-66641de0607c476a958313ea66dff4db on tablet 66641de0607c476a958313ea66dff4db with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:38.672204 2006 tablet_copy_source_session.cc:205] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:38.677541 2717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:38.683404 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet 91be317234ca451fad76824aef0861a6 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:21:38.684001 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet 91be317234ca451fad76824aef0861a6 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-91be317234ca451fad76824aef0861a6
I20260924 00:21:38.693506 2717 tablet_copy_client.cc:791] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:38.694160 2717 tablet_copy_client.cc:655] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:38.695816 2155 tablet_copy_source_session.cc:205] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:38.701949 2717 tablet_copy_client.cc:531] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:38.703008 2714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91be317234ca451fad76824aef0861a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:38.713958 2717 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:38.721235 2714 tablet_copy_client.cc:791] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:38.721858 2714 tablet_copy_client.cc:655] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:38.725566 2714 tablet_copy_client.cc:531] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:38.733811 2717 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:38.734182 2714 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:38.734673 2717 tablet_bootstrap.cc:492] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:38.735416 2717 ts_tablet_manager.cc:1397] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20260924 00:21:38.738332 2717 raft_consensus.cc:348] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.739286 2717 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:38.739952 2717 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.745111 2717 ts_tablet_manager.cc:1428] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260924 00:21:38.748068 2006 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-6cb231d6aeb14f4d83ab20c17ea2f9e1 received from {username='slave'} at 127.0.0.1:42938
I20260924 00:21:38.748481 2006 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-6cb231d6aeb14f4d83ab20c17ea2f9e1 on tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:38.761438 2714 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:38.762396 2714 tablet_bootstrap.cc:492] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:38.765741 2714 ts_tablet_manager.cc:1397] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.006s
I20260924 00:21:38.768982 2714 raft_consensus.cc:348] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.770196 2714 raft_consensus.cc:729] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:38.771281 2714 consensus_queue.cc:260] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:38.773510 2714 ts_tablet_manager.cc:1428] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 00:21:38.775692 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-91be317234ca451fad76824aef0861a6 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:21:38.776077 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-91be317234ca451fad76824aef0861a6 on tablet 91be317234ca451fad76824aef0861a6 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:38.877995 2649 raft_consensus.cc:1203] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:38.997462 2649 raft_consensus.cc:1203] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:39.062783 2649 raft_consensus.cc:1203] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260924 00:21:39.164696 2649 raft_consensus.cc:1203] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:39.321018 2515 raft_consensus.cc:1050] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:39.323511 2515 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.331720 2141 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.332487 2066 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.333557 2649 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.335428 2684 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.337224 2684 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.342273 2515 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:39.365060 2684 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.369968 2649 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.377444 2141 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.390225 2722 raft_consensus.cc:1050] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:39.389703 2066 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.390654 1892 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.392174 2722 consensus_queue.cc:237] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.417264 2649 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.417393 2139 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.422281 2684 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.427508 2515 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:39.428720 2066 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.433425 2515 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.458395 2515 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.464025 2141 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.465766 1993 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.474758 2068 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.480707 2649 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:39.481810 2648 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.488607 2141 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:39.487968 1893 catalog_manager.cc:5495] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.491324 2722 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:39.493393 2515 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:39.513458 2141 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.514405 2648 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.515543 2515 raft_consensus.cc:2932] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.528404 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 217f6c4b9f5c44d0b3856a8d131557cd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:39.537408 1991 consensus_queue.cc:237] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.544922 2648 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:39.545058 2068 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:39.546525 2515 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:39.545107 1892 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:39.548731 2684 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:39.563546 2684 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.575085 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 217f6c4b9f5c44d0b3856a8d131557cd with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47762
I20260924 00:21:39.580022 2730 raft_consensus.cc:1050] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:39.581766 2737 tablet_replica.cc:330] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:21:39.582820 2737 raft_consensus.cc:2229] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:39.582617 2730 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.583585 2737 raft_consensus.cc:2258] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:39.586788 2737 ts_tablet_manager.cc:1905] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:39.593701 2648 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.595113 2068 raft_consensus.cc:2932] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.603716 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 66641de0607c476a958313ea66dff4db with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:39.608021 2649 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 00:21:39.608089 1893 catalog_manager.cc:5495] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:39.617559 2737 ts_tablet_manager.cc:1918] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:39.618088 2737 log.cc:1196] T 217f6c4b9f5c44d0b3856a8d131557cd P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/217f6c4b9f5c44d0b3856a8d131557cd
I20260924 00:21:39.620086 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet 217f6c4b9f5c44d0b3856a8d131557cd (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:39.628810 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 66641de0607c476a958313ea66dff4db with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57740
I20260924 00:21:39.630863 2068 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 00:21:39.632673 2141 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 00:21:39.632709 2730 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:39.639315 2730 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.641495 2722 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.655016 2739 tablet_replica.cc:330] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:21:39.656176 2739 raft_consensus.cc:2229] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:39.657827 2739 raft_consensus.cc:2258] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:39.658871 2723 raft_consensus.cc:1050] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:39.663985 2739 ts_tablet_manager.cc:1905] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:39.667097 2723 consensus_queue.cc:237] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.672703 2684 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.679226 2648 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.679488 2141 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.686966 1991 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.687587 2649 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.689324 2723 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.689946 2068 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:39.691092 2694 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.694500 2694 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:39.707253 2066 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.717865 1893 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.720579 2739 ts_tablet_manager.cc:1918] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:39.721028 2739 log.cc:1196] T 66641de0607c476a958313ea66dff4db P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/66641de0607c476a958313ea66dff4db
I20260924 00:21:39.729691 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 66641de0607c476a958313ea66dff4db (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:39.745887 2694 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.748875 1991 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.750240 2648 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.754261 1993 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.757068 2066 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.764572 2143 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260924 00:21:39.766705 2648 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260924 00:21:39.767704 2684 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:39.768781 2515 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:39.769444 1893 catalog_manager.cc:5495] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.779801 2730 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.781780 2140 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.783152 2649 raft_consensus.cc:2932] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.794205 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:39.809725 2143 consensus_queue.cc:237] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:39.810979 1893 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 2 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.817341 2068 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:39.818776 2693 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:39.824602 2649 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:39.825780 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47762
I20260924 00:21:39.826428 2693 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:39.826668 2737 tablet_replica.cc:330] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:21:39.827629 2737 raft_consensus.cc:2229] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:39.828320 2737 raft_consensus.cc:2258] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:39.831683 2737 ts_tablet_manager.cc:1905] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:39.843633 2694 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.853924 2737 ts_tablet_manager.cc:1918] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260924 00:21:39.854437 2737 log.cc:1196] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/6cb231d6aeb14f4d83ab20c17ea2f9e1
I20260924 00:21:39.856037 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet 6cb231d6aeb14f4d83ab20c17ea2f9e1 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:39.855926 2068 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.860589 2649 raft_consensus.cc:2932] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:39.861707 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 91be317234ca451fad76824aef0861a6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:39.868738 1890 catalog_manager.cc:5495] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:39.879252 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet 91be317234ca451fad76824aef0861a6 with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260924 00:21:39.882776 2748 tablet_replica.cc:330] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:21:39.883849 2748 raft_consensus.cc:2229] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:39.884547 2748 raft_consensus.cc:2258] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:39.888144 2748 ts_tablet_manager.cc:1905] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:39.904314 2748 ts_tablet_manager.cc:1918] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:39.904807 2748 log.cc:1196] T 91be317234ca451fad76824aef0861a6 P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/91be317234ca451fad76824aef0861a6
I20260924 00:21:39.906638 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet 91be317234ca451fad76824aef0861a6 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:40.948388 2068 consensus_queue.cc:237] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:40.959198 1991 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:40.961265 2750 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:40.961819 2141 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:40.974222 2750 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:40.990648 2384 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:40.998263 2143 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.008953 1893 catalog_manager.cc:5495] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:41.012005 2141 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:41.013875 2035 consensus_peers.cc:487] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93bfcd566e3e4c77b3a672d5705a546f. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:41.023519 1991 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.032853 2068 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 00:21:41.035760 2109 consensus_peers.cc:487] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e92f7a727fb4c1da7807d1429d98861. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:41.037395 2749 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:41.038817 1992 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:41.041287 2693 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:41.061183 2723 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.065690 2066 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.078502 2068 consensus_queue.cc:237] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.076944 1992 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.109155 1890 catalog_manager.cc:5495] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
W20260924 00:21:41.117803 2035 consensus_peers.cc:487] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fa219f0283c4632b6601e60980ae25a. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:41.117856 2141 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:41.119751 2750 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:41.121521 1992 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:41.122862 2750 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:41.131237 2692 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.134194 2141 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.133738 1992 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.143471 2068 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:41.155951 2035 consensus_peers.cc:487] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c616f076fa3a4e75b7d3c51e60553d74. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:41.159605 2140 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:41.159952 1992 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:41.162109 2692 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:41.162804 2750 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:41.164439 1890 catalog_manager.cc:5495] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.199048 2692 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.205461 2140 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.218360 1992 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:41.233448 1892 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:41.420374 2773 ts_tablet_manager.cc:927] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:41.422397 2773 tablet_copy_client.cc:316] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:21:41.423723 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet 8e92f7a727fb4c1da7807d1429d98861 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:21:41.424206 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet 8e92f7a727fb4c1da7807d1429d98861 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-8e92f7a727fb4c1da7807d1429d98861
I20260924 00:21:41.430968 2155 tablet_copy_source_session.cc:205] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:41.434244 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e92f7a727fb4c1da7807d1429d98861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:41.449894 2773 tablet_copy_client.cc:791] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:41.450486 2773 tablet_copy_client.cc:655] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:41.456140 2773 tablet_copy_client.cc:531] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:41.464819 2773 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:41.483361 2773 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:41.484055 2773 tablet_bootstrap.cc:492] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:41.484702 2773 ts_tablet_manager.cc:1397] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.002s
I20260924 00:21:41.486851 2773 raft_consensus.cc:348] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.487591 2773 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:41.488205 2773 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.491211 2773 ts_tablet_manager.cc:1428] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
I20260924 00:21:41.492661 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-8e92f7a727fb4c1da7807d1429d98861 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:21:41.493037 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-8e92f7a727fb4c1da7807d1429d98861 on tablet 8e92f7a727fb4c1da7807d1429d98861 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:41.517103 2773 ts_tablet_manager.cc:927] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:41.518611 2773 tablet_copy_client.cc:316] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:41.546847 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 93bfcd566e3e4c77b3a672d5705a546f from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:41.547382 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 93bfcd566e3e4c77b3a672d5705a546f from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-93bfcd566e3e4c77b3a672d5705a546f
I20260924 00:21:41.555032 2081 tablet_copy_source_session.cc:205] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:41.557708 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bfcd566e3e4c77b3a672d5705a546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:41.574379 2773 tablet_copy_client.cc:791] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:41.574913 2773 tablet_copy_client.cc:655] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:41.578542 2773 tablet_copy_client.cc:531] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:41.586921 2773 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:41.605465 2773 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:41.606367 2773 tablet_bootstrap.cc:492] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:41.607054 2773 ts_tablet_manager.cc:1397] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20260924 00:21:41.609807 2773 raft_consensus.cc:348] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.610759 2773 raft_consensus.cc:729] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:41.611465 2773 consensus_queue.cc:260] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.613925 2773 ts_tablet_manager.cc:1428] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 00:21:41.618479 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-93bfcd566e3e4c77b3a672d5705a546f received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:41.618975 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-93bfcd566e3e4c77b3a672d5705a546f on tablet 93bfcd566e3e4c77b3a672d5705a546f with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:41.681790 2773 ts_tablet_manager.cc:927] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:41.684150 2773 tablet_copy_client.cc:316] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:41.686031 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet c616f076fa3a4e75b7d3c51e60553d74 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:41.686592 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet c616f076fa3a4e75b7d3c51e60553d74 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-c616f076fa3a4e75b7d3c51e60553d74
I20260924 00:21:41.694700 2081 tablet_copy_source_session.cc:205] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:41.697506 2773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c616f076fa3a4e75b7d3c51e60553d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:41.709837 2773 tablet_copy_client.cc:791] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:41.710414 2773 tablet_copy_client.cc:655] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:41.712330 2783 ts_tablet_manager.cc:927] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:41.714707 2773 tablet_copy_client.cc:531] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:41.719790 2783 tablet_copy_client.cc:316] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:41.721417 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 7fa219f0283c4632b6601e60980ae25a from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:41.721796 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 7fa219f0283c4632b6601e60980ae25a from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-7fa219f0283c4632b6601e60980ae25a
I20260924 00:21:41.726599 2773 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:41.730533 2081 tablet_copy_source_session.cc:205] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:41.735466 2783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa219f0283c4632b6601e60980ae25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:41.758414 2783 tablet_copy_client.cc:791] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:41.758968 2783 tablet_copy_client.cc:655] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:41.760758 2773 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:41.761651 2773 tablet_bootstrap.cc:492] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:41.762838 2773 ts_tablet_manager.cc:1397] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.012s
I20260924 00:21:41.763878 2783 tablet_copy_client.cc:531] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:41.765789 2773 raft_consensus.cc:348] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.766721 2773 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:41.767288 2773 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.769568 2773 ts_tablet_manager.cc:1428] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260924 00:21:41.784657 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-c616f076fa3a4e75b7d3c51e60553d74 received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:41.785106 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-c616f076fa3a4e75b7d3c51e60553d74 on tablet c616f076fa3a4e75b7d3c51e60553d74 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:41.794862 2783 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:41.815223 2783 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:41.816112 2783 tablet_bootstrap.cc:492] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:41.816702 2783 ts_tablet_manager.cc:1397] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20260924 00:21:41.819043 2783 raft_consensus.cc:348] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.819977 2783 raft_consensus.cc:729] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:41.820611 2783 consensus_queue.cc:260] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:41.823341 2783 ts_tablet_manager.cc:1428] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260924 00:21:41.829871 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-7fa219f0283c4632b6601e60980ae25a received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:41.830296 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-7fa219f0283c4632b6601e60980ae25a on tablet 7fa219f0283c4632b6601e60980ae25a with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:41.937616 2649 raft_consensus.cc:1203] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:41.989967 2649 raft_consensus.cc:1203] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:42.107009 2649 raft_consensus.cc:1203] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:42.141494 2649 raft_consensus.cc:1203] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:42.381433 2723 raft_consensus.cc:1050] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:42.383733 2723 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:42.391664 2649 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.392807 1991 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.392886 2066 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.394277 2787 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:42.399705 2789 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:42.400568 2787 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:42.429041 2789 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.430670 2649 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.430743 2066 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.442956 1892 catalog_manager.cc:5495] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.449968 1991 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.484683 2141 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:42.488951 2649 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:42.489440 2068 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:42.490907 2789 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:42.491796 2788 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:42.498430 2788 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.506350 2649 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.509709 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8e92f7a727fb4c1da7807d1429d98861 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:42.516844 2692 raft_consensus.cc:1050] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:42.518193 1891 catalog_manager.cc:5495] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:42.519003 2692 consensus_queue.cc:237] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:42.527262 2648 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.529022 2750 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:42.534523 1991 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.539438 2758 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:42.550535 2143 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.551813 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e92f7a727fb4c1da7807d1429d98861 with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260924 00:21:42.552398 2758 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:42.552313 2066 raft_consensus.cc:2932] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.568189 2692 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.569881 1991 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.582844 2649 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.595945 2802 tablet_replica.cc:330] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:21:42.602602 1892 catalog_manager.cc:5495] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.619966 2802 raft_consensus.cc:2229] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:42.620673 2802 raft_consensus.cc:2258] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:42.625619 2802 ts_tablet_manager.cc:1905] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:42.630743 2063 consensus_queue.cc:237] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:42.638357 2649 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:42.638907 1992 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:42.641331 2804 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:42.642886 2804 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260924 00:21:42.650440 2692 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.657233 2802 ts_tablet_manager.cc:1918] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:42.657680 2802 log.cc:1196] T 8e92f7a727fb4c1da7807d1429d98861 P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/8e92f7a727fb4c1da7807d1429d98861
I20260924 00:21:42.661257 2648 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.663436 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet 8e92f7a727fb4c1da7807d1429d98861 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:42.666630 2804 raft_consensus.cc:1050] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:42.666236 1992 raft_consensus.cc:2932] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.671160 2804 consensus_queue.cc:237] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:42.686663 2648 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.686113 1892 catalog_manager.cc:5495] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.687352 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 93bfcd566e3e4c77b3a672d5705a546f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:42.688539 2750 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:42.707798 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 93bfcd566e3e4c77b3a672d5705a546f with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57740
I20260924 00:21:42.710552 2808 tablet_replica.cc:330] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:21:42.712831 2143 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.717316 2805 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:42.712874 2808 raft_consensus.cc:2229] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:42.722014 2808 pending_rounds.cc:62] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Trying to abort 1 pending ops.
I20260924 00:21:42.722577 1992 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.722333 2808 pending_rounds.cc:68] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7332697303112957952 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } } }
I20260924 00:21:42.723744 2808 raft_consensus.cc:2894] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260924 00:21:42.724175 2808 raft_consensus.cc:2866] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260924 00:21:42.724545 2808 raft_consensus.cc:2258] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:42.725745 2750 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:42.733199 2808 ts_tablet_manager.cc:1905] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:42.734022 2758 raft_consensus.cc:1050] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:42.737006 2758 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:42.739375 2804 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.742735 1992 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.748358 2648 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.750408 2140 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.752118 2758 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:42.754676 2750 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:42.758272 2648 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.764137 1891 catalog_manager.cc:5495] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.771124 2143 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.795078 2808 ts_tablet_manager.cc:1918] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:42.795544 2808 log.cc:1196] T 93bfcd566e3e4c77b3a672d5705a546f P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/93bfcd566e3e4c77b3a672d5705a546f
I20260924 00:21:42.797511 2804 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.797364 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 93bfcd566e3e4c77b3a672d5705a546f (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:42.811376 2143 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.813685 1992 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:42.815272 2750 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:42.814280 2648 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.821163 1992 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.823092 2063 consensus_queue.cc:237] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:42.840502 1891 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.857614 2648 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:42.860687 2805 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:42.860913 2143 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:42.862366 2750 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:42.872478 2750 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.874145 2649 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.881187 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7fa219f0283c4632b6601e60980ae25a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:42.885038 2143 raft_consensus.cc:2932] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.895669 1890 catalog_manager.cc:5495] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:42.918031 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fa219f0283c4632b6601e60980ae25a with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260924 00:21:42.918943 2802 tablet_replica.cc:330] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:21:42.919970 2802 raft_consensus.cc:2229] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:42.920619 2802 raft_consensus.cc:2258] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:42.923727 2802 ts_tablet_manager.cc:1905] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:42.941499 2802 ts_tablet_manager.cc:1918] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:42.941819 2802 log.cc:1196] T 7fa219f0283c4632b6601e60980ae25a P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/7fa219f0283c4632b6601e60980ae25a
I20260924 00:21:42.942992 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet 7fa219f0283c4632b6601e60980ae25a (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:43.268699 2066 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:43.272851 1992 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:43.272851 2649 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:43.274223 2804 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:43.275151 2805 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:43.280998 2804 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:43.286356 2649 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:43.287931 1992 raft_consensus.cc:2932] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:43.298861 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c616f076fa3a4e75b7d3c51e60553d74 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:43.306252 1890 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:43.316881 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet c616f076fa3a4e75b7d3c51e60553d74 with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57740
I20260924 00:21:43.319772 2817 tablet_replica.cc:330] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:21:43.320644 2817 raft_consensus.cc:2229] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:43.321244 2817 raft_consensus.cc:2258] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:43.324301 2817 ts_tablet_manager.cc:1905] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:43.335459 2817 ts_tablet_manager.cc:1918] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:43.335896 2817 log.cc:1196] T c616f076fa3a4e75b7d3c51e60553d74 P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/c616f076fa3a4e75b7d3c51e60553d74
I20260924 00:21:43.337388 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet c616f076fa3a4e75b7d3c51e60553d74 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:44.371091 1991 consensus_queue.cc:237] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:44.377923 2066 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 00:21:44.377912 1960 consensus_peers.cc:487] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d581036c5b6e448cbae056fa4fc15f78. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:44.378201 2141 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:44.379287 2816 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:44.380200 2730 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:44.391709 2816 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.398685 2066 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.399981 2143 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.404366 2068 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:44.409893 1892 catalog_manager.cc:5495] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260924 00:21:44.432063 2035 consensus_peers.cc:487] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b76aee4623d7484e8ff6c1f1a13f5fea. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:44.432379 2143 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:44.434160 2750 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:44.436247 1991 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:44.437705 2804 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:44.446460 2818 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.449066 1991 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.449267 2141 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.469982 1991 consensus_queue.cc:237] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:44.472877 1891 catalog_manager.cc:5495] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.498378 2141 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 00:21:44.500415 1960 consensus_peers.cc:487] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaea92f011f94575ab225f58d2a0db06. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:44.500581 2068 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:44.500756 2821 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:44.502758 2821 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:44.523269 2821 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.527822 2140 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.532135 2068 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.544680 1891 catalog_manager.cc:5495] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.547145 2066 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:44.568385 2035 consensus_peers.cc:487] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a1695e8b6d44e39a87ae6431db2eb9b. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:44.569088 2141 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:44.569833 1991 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:44.570814 2750 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:44.572340 2818 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:44.578735 2750 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.580581 2141 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.584259 1991 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.597474 1891 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:44.893225 2839 ts_tablet_manager.cc:927] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:44.895304 2839 tablet_copy_client.cc:316] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
I20260924 00:21:44.896943 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet aaea92f011f94575ab225f58d2a0db06 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:21:44.897658 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet aaea92f011f94575ab225f58d2a0db06 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-aaea92f011f94575ab225f58d2a0db06
I20260924 00:21:44.907296 2006 tablet_copy_source_session.cc:205] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:44.911355 2839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaea92f011f94575ab225f58d2a0db06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:44.925482 2839 tablet_copy_client.cc:791] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:44.926092 2839 tablet_copy_client.cc:655] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:44.929804 2839 tablet_copy_client.cc:531] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:44.939286 2839 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:44.974839 2841 ts_tablet_manager.cc:927] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:44.981504 2841 tablet_copy_client.cc:316] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:44.983114 2839 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:44.983659 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet b76aee4623d7484e8ff6c1f1a13f5fea from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:44.984063 2839 tablet_bootstrap.cc:492] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:44.984220 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet b76aee4623d7484e8ff6c1f1a13f5fea from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-b76aee4623d7484e8ff6c1f1a13f5fea
I20260924 00:21:44.984781 2839 ts_tablet_manager.cc:1397] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.046s user 0.025s sys 0.004s
I20260924 00:21:44.987538 2839 raft_consensus.cc:348] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:44.988461 2839 raft_consensus.cc:729] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:44.989069 2839 consensus_queue.cc:260] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:44.992703 2845 ts_tablet_manager.cc:927] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:44.993186 2839 ts_tablet_manager.cc:1428] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260924 00:21:44.995205 2006 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-aaea92f011f94575ab225f58d2a0db06 received from {username='slave'} at 127.0.0.1:42938
I20260924 00:21:44.995601 2006 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-aaea92f011f94575ab225f58d2a0db06 on tablet aaea92f011f94575ab225f58d2a0db06 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:45.005304 2081 tablet_copy_source_session.cc:205] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:45.005841 2845 tablet_copy_client.cc:316] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
I20260924 00:21:45.007752 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet d581036c5b6e448cbae056fa4fc15f78 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:21:45.008195 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet d581036c5b6e448cbae056fa4fc15f78 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-d581036c5b6e448cbae056fa4fc15f78
I20260924 00:21:45.017315 2841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76aee4623d7484e8ff6c1f1a13f5fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:45.019877 2006 tablet_copy_source_session.cc:205] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:45.024181 2845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d581036c5b6e448cbae056fa4fc15f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:45.034456 2841 tablet_copy_client.cc:791] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:45.035056 2841 tablet_copy_client.cc:655] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:45.043097 2841 tablet_copy_client.cc:531] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:45.055671 2839 ts_tablet_manager.cc:927] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:45.062085 2841 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:45.064610 2845 tablet_copy_client.cc:791] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:45.067205 2845 tablet_copy_client.cc:655] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:45.073535 2839 tablet_copy_client.cc:316] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:45.075412 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 4a1695e8b6d44e39a87ae6431db2eb9b from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:45.075909 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 4a1695e8b6d44e39a87ae6431db2eb9b from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-4a1695e8b6d44e39a87ae6431db2eb9b
I20260924 00:21:45.076454 2845 tablet_copy_client.cc:531] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:45.087255 2081 tablet_copy_source_session.cc:205] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:45.090334 2839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1695e8b6d44e39a87ae6431db2eb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:45.092471 2845 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:45.101605 2841 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:45.102766 2841 tablet_bootstrap.cc:492] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:45.103480 2841 ts_tablet_manager.cc:1397] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.042s user 0.025s sys 0.002s
I20260924 00:21:45.108156 2841 raft_consensus.cc:348] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:45.109040 2841 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:45.109740 2841 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:45.114878 2841 ts_tablet_manager.cc:1428] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260924 00:21:45.120090 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-b76aee4623d7484e8ff6c1f1a13f5fea received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:45.120527 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-b76aee4623d7484e8ff6c1f1a13f5fea on tablet b76aee4623d7484e8ff6c1f1a13f5fea with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:45.135856 2839 tablet_copy_client.cc:791] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:45.136650 2839 tablet_copy_client.cc:655] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:45.152829 2839 tablet_copy_client.cc:531] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:45.157972 2845 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:45.158927 2845 tablet_bootstrap.cc:492] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:45.159698 2845 ts_tablet_manager.cc:1397] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.068s user 0.027s sys 0.004s
I20260924 00:21:45.160866 2839 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:45.163328 2845 raft_consensus.cc:348] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:45.164628 2845 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:45.165372 2845 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:45.167761 2845 ts_tablet_manager.cc:1428] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 00:21:45.202461 2006 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-d581036c5b6e448cbae056fa4fc15f78 received from {username='slave'} at 127.0.0.1:42938
I20260924 00:21:45.204034 2006 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-d581036c5b6e448cbae056fa4fc15f78 on tablet d581036c5b6e448cbae056fa4fc15f78 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:45.286145 2839 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:45.287358 2839 tablet_bootstrap.cc:492] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:45.288774 2839 ts_tablet_manager.cc:1397] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.128s user 0.021s sys 0.012s
I20260924 00:21:45.294204 2839 raft_consensus.cc:348] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:45.296186 2839 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:45.297158 2839 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:45.301285 2839 ts_tablet_manager.cc:1428] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260924 00:21:45.337639 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-4a1695e8b6d44e39a87ae6431db2eb9b received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:45.338343 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-4a1695e8b6d44e39a87ae6431db2eb9b on tablet 4a1695e8b6d44e39a87ae6431db2eb9b with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:45.367386 2648 raft_consensus.cc:1203] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:45.392333 2648 raft_consensus.cc:1203] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:45.471207 2648 raft_consensus.cc:1203] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:45.480242 2648 raft_consensus.cc:1203] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:45.817365 2851 raft_consensus.cc:1050] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:45.820729 2851 consensus_queue.cc:237] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:45.834537 2648 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.835088 2139 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.835721 2066 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.838459 2730 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:45.840888 2730 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:45.845223 2816 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:45.862542 2750 raft_consensus.cc:1050] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:45.866709 2750 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:45.872785 2857 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.875187 2068 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.896152 2648 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.897881 2852 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:45.910559 2649 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.914605 2140 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.917371 2852 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:45.943404 2855 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.944721 2143 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.950997 1890 catalog_manager.cc:5495] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.965266 1995 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.967495 1892 catalog_manager.cc:5495] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.979179 2816 raft_consensus.cc:1050] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:45.978672 2649 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.981724 2816 consensus_queue.cc:237] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:45.986788 2855 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:45.992666 2139 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:45.996971 2143 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.997536 2648 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:45.998672 2851 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:46.004433 2851 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:46.009327 2068 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:46.014643 1991 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.019613 2816 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:46.044099 2852 raft_consensus.cc:1050] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:46.048894 2852 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:46.055315 2730 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.060113 2139 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.098392 1890 catalog_manager.cc:5495] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:46.076449 2648 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.100484 2068 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:46.105029 2143 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:46.112044 2649 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:46.090492 2065 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.137311 2876 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:46.141474 2878 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:46.148041 1991 consensus_queue.cc:237] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:46.167819 1992 consensus_queue.cc:237] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:46.176302 2649 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.179076 2816 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:46.179486 2143 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.183398 2816 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:46.184389 2647 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.186807 2143 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.187505 2857 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:46.219520 2649 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.207796 2816 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.212298 2139 raft_consensus.cc:2932] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.205122 2857 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260924 00:21:46.244189 1890 catalog_manager.cc:5495] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.253707 2851 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.254820 2648 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
W20260924 00:21:46.263633 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 86458.5us
I20260924 00:21:46.264679 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d581036c5b6e448cbae056fa4fc15f78 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:46.269801 2140 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.272909 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet aaea92f011f94575ab225f58d2a0db06 with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47762
I20260924 00:21:46.281278 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet aaea92f011f94575ab225f58d2a0db06 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260924 00:21:46.293440 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 134518us
I20260924 00:21:46.296352 1995 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:46.305063 2646 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.331116 2852 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:46.334501 1890 catalog_manager.cc:5495] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.325567 2145 raft_consensus.cc:2932] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.366689 2884 tablet_replica.cc:330] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:21:46.368249 2884 raft_consensus.cc:2229] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:46.379778 2882 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260924 00:21:46.369084 2884 raft_consensus.cc:2258] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:46.418745 2884 ts_tablet_manager.cc:1905] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:46.413126 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet d581036c5b6e448cbae056fa4fc15f78 with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47762
I20260924 00:21:46.432747 2879 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.445657 2145 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.450834 2649 raft_consensus.cc:2932] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.459424 2880 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:46.464902 2750 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.470151 2647 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.482291 1890 catalog_manager.cc:5495] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.489173 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b76aee4623d7484e8ff6c1f1a13f5fea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:46.488706 2145 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.511045 1893 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:46.515909 2884 ts_tablet_manager.cc:1918] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:46.516703 2884 log.cc:1196] T aaea92f011f94575ab225f58d2a0db06 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/aaea92f011f94575ab225f58d2a0db06
I20260924 00:21:46.520380 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet aaea92f011f94575ab225f58d2a0db06 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:46.521540 2884 tablet_replica.cc:330] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:21:46.523373 2884 raft_consensus.cc:2229] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:46.525177 2884 raft_consensus.cc:2258] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:46.529538 2884 ts_tablet_manager.cc:1905] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:46.547585 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet b76aee4623d7484e8ff6c1f1a13f5fea with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260924 00:21:46.554751 2884 ts_tablet_manager.cc:1918] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:46.556501 2884 log.cc:1196] T d581036c5b6e448cbae056fa4fc15f78 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/d581036c5b6e448cbae056fa4fc15f78
I20260924 00:21:46.558959 2889 tablet_replica.cc:330] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:21:46.561619 1991 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.564930 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet d581036c5b6e448cbae056fa4fc15f78 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:46.574718 2065 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:46.581810 2889 raft_consensus.cc:2229] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:46.582533 2889 raft_consensus.cc:2258] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:46.585346 2647 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.586308 2889 ts_tablet_manager.cc:1905] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:46.586959 2879 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:46.594798 2750 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.603479 2889 ts_tablet_manager.cc:1918] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:46.604007 2889 log.cc:1196] T b76aee4623d7484e8ff6c1f1a13f5fea P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/b76aee4623d7484e8ff6c1f1a13f5fea
I20260924 00:21:46.604919 2647 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.607095 1991 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:46.609383 2879 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:46.610741 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet b76aee4623d7484e8ff6c1f1a13f5fea (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:46.614995 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4a1695e8b6d44e39a87ae6431db2eb9b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:46.617247 1992 raft_consensus.cc:2932] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:46.631057 1892 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:46.642316 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a1695e8b6d44e39a87ae6431db2eb9b with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57740
I20260924 00:21:46.644945 2890 tablet_replica.cc:330] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:21:46.646003 2890 raft_consensus.cc:2229] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:46.646993 2890 raft_consensus.cc:2258] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:46.650377 2890 ts_tablet_manager.cc:1905] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:46.665555 2890 ts_tablet_manager.cc:1918] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:46.665956 2890 log.cc:1196] T 4a1695e8b6d44e39a87ae6431db2eb9b P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/4a1695e8b6d44e39a87ae6431db2eb9b
I20260924 00:21:46.669049 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 4a1695e8b6d44e39a87ae6431db2eb9b (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
W20260924 00:21:47.663681 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 79930.2us
W20260924 00:21:47.955296 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 52630.5us
I20260924 00:21:47.989517 2062 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:48.086810 2035 consensus_peers.cc:487] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 447705637cdc44668a870a52a7a3871a. This is attempt 1: this message will repeat every 5th retry.
W20260924 00:21:48.091428 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 133240us
I20260924 00:21:48.100907 1992 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:48.124204 2145 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:48.132054 2750 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:48.143918 2893 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:48.167011 2750 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.169755 2143 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.179584 2141 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.188789 1893 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:48.202512 2067 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:48.203322 1996 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:48.206223 1996 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.208078 2789 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260924 00:21:48.212054 2109 consensus_peers.cc:487] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f75d53c3ccf437cac00ba5d83b2c165. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:48.212510 2896 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:48.237010 2873 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.238945 2063 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.245594 1994 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.268903 1996 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.283816 1892 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.291887 2143 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 00:21:48.294046 1960 consensus_peers.cc:487] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92bd85e8dadb4876827acd9ff4be377f. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:48.295148 2063 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:48.299094 2900 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:48.303012 2895 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:48.318336 2895 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.326592 2063 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.340967 2067 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.345214 2143 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.357452 1893 catalog_manager.cc:5495] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
W20260924 00:21:48.373687 2035 consensus_peers.cc:487] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ff940e15e1849efb194c908520c212f. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:48.374703 1996 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:48.375640 2143 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:48.376281 2750 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:48.377739 2879 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:48.404841 2894 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.407333 2143 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.416316 1996 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:48.439690 1893 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:48.615031 2919 ts_tablet_manager.cc:927] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:48.624450 2919 tablet_copy_client.cc:316] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:48.626014 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 447705637cdc44668a870a52a7a3871a from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:48.626569 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 447705637cdc44668a870a52a7a3871a from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-447705637cdc44668a870a52a7a3871a
I20260924 00:21:48.633561 2081 tablet_copy_source_session.cc:205] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:48.636998 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447705637cdc44668a870a52a7a3871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:48.652338 2919 tablet_copy_client.cc:791] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:48.654922 2919 tablet_copy_client.cc:655] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:48.659248 2919 tablet_copy_client.cc:531] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:48.668246 2919 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:48.689339 2919 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:48.690289 2919 tablet_bootstrap.cc:492] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:48.691008 2919 ts_tablet_manager.cc:1397] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20260924 00:21:48.694316 2919 raft_consensus.cc:348] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.695348 2919 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:48.696012 2919 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.699856 2919 ts_tablet_manager.cc:1428] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260924 00:21:48.701687 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-447705637cdc44668a870a52a7a3871a received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:48.702036 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-447705637cdc44668a870a52a7a3871a on tablet 447705637cdc44668a870a52a7a3871a with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:48.710243 2919 ts_tablet_manager.cc:927] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:48.723934 2919 tablet_copy_client.cc:316] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
I20260924 00:21:48.725630 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet 92bd85e8dadb4876827acd9ff4be377f from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:21:48.726140 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet 92bd85e8dadb4876827acd9ff4be377f from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-92bd85e8dadb4876827acd9ff4be377f
I20260924 00:21:48.731741 2006 tablet_copy_source_session.cc:205] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:48.734426 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bd85e8dadb4876827acd9ff4be377f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:48.747411 2919 tablet_copy_client.cc:791] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:48.748008 2919 tablet_copy_client.cc:655] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:48.752252 2919 tablet_copy_client.cc:531] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:48.761111 2919 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:48.780195 2919 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:48.781121 2919 tablet_bootstrap.cc:492] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:48.781919 2919 ts_tablet_manager.cc:1397] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.008s
I20260924 00:21:48.784922 2919 raft_consensus.cc:348] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.785867 2919 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:48.786516 2919 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.788647 2919 ts_tablet_manager.cc:1428] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260924 00:21:48.791633 2006 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-92bd85e8dadb4876827acd9ff4be377f received from {username='slave'} at 127.0.0.1:42938
I20260924 00:21:48.792050 2006 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-92bd85e8dadb4876827acd9ff4be377f on tablet 92bd85e8dadb4876827acd9ff4be377f with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:48.818886 2919 ts_tablet_manager.cc:927] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:48.822357 2919 tablet_copy_client.cc:316] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:21:48.824775 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet 2f75d53c3ccf437cac00ba5d83b2c165 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:21:48.825310 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet 2f75d53c3ccf437cac00ba5d83b2c165 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-2f75d53c3ccf437cac00ba5d83b2c165
I20260924 00:21:48.836454 2155 tablet_copy_source_session.cc:205] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:48.839763 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f75d53c3ccf437cac00ba5d83b2c165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:48.855892 2919 tablet_copy_client.cc:791] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:48.856477 2919 tablet_copy_client.cc:655] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:48.866806 2919 tablet_copy_client.cc:531] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:48.884761 2919 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:48.905190 2919 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:48.906248 2919 tablet_bootstrap.cc:492] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:48.906860 2919 ts_tablet_manager.cc:1397] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.004s
I20260924 00:21:48.909566 2919 raft_consensus.cc:348] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.910351 2919 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:48.910938 2919 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:48.915823 2919 ts_tablet_manager.cc:1428] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260924 00:21:48.917318 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-2f75d53c3ccf437cac00ba5d83b2c165 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:21:48.917730 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-2f75d53c3ccf437cac00ba5d83b2c165 on tablet 2f75d53c3ccf437cac00ba5d83b2c165 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:48.993325 2919 ts_tablet_manager.cc:927] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:48.994724 2919 tablet_copy_client.cc:316] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:48.996073 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 9ff940e15e1849efb194c908520c212f from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:48.997061 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 9ff940e15e1849efb194c908520c212f from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-9ff940e15e1849efb194c908520c212f
I20260924 00:21:49.004285 2081 tablet_copy_source_session.cc:205] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:49.007201 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff940e15e1849efb194c908520c212f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 00:21:49.016585 2017 debug-util.cc:398] Leaking SignalData structure 0x7b0800182dc0 after lost signal to thread 815
W20260924 00:21:49.017679 2017 debug-util.cc:398] Leaking SignalData structure 0x7b0800182d20 after lost signal to thread 1928
W20260924 00:21:49.018795 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08002910a0 after lost signal to thread 2020
W20260924 00:21:49.019769 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08002f1840 after lost signal to thread 2095
W20260924 00:21:49.020773 2017 debug-util.cc:398] Leaking SignalData structure 0x7b080029b820 after lost signal to thread 2169
W20260924 00:21:49.021797 2017 debug-util.cc:398] Leaking SignalData structure 0x7b0800289160 after lost signal to thread 2673
I20260924 00:21:49.022152 2919 tablet_copy_client.cc:791] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:49.022701 2919 tablet_copy_client.cc:655] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:49.029210 2919 tablet_copy_client.cc:531] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:49.037313 2919 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:49.057173 2649 raft_consensus.cc:1203] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:49.060806 2919 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:49.061694 2919 tablet_bootstrap.cc:492] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:49.062310 2919 ts_tablet_manager.cc:1397] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.000s
I20260924 00:21:49.064343 2919 raft_consensus.cc:348] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:49.065078 2919 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:49.065650 2919 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:49.068749 2919 ts_tablet_manager.cc:1428] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 00:21:49.070544 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-9ff940e15e1849efb194c908520c212f received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:49.071339 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-9ff940e15e1849efb194c908520c212f on tablet 9ff940e15e1849efb194c908520c212f with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:49.193585 2649 raft_consensus.cc:1203] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:49.281564 2649 raft_consensus.cc:1203] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:49.503157 2894 raft_consensus.cc:1050] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:49.505443 2894 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:49.527591 2649 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.528908 2143 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.529157 2894 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:49.530452 2893 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:49.534758 1994 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.540740 2879 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:49.542299 2647 raft_consensus.cc:1203] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:49.604043 2879 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.608951 2145 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.614588 1994 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.639060 2649 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.644550 1893 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.740093 2067 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:49.749514 2895 raft_consensus.cc:1050] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:49.752439 2895 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:49.759070 2649 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:49.762758 2939 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:49.770440 2933 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.779911 2647 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.793772 2647 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.795034 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 447705637cdc44668a870a52a7a3871a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:49.796090 2145 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.795590 1893 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.798563 2067 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.800092 1994 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:49.802204 2938 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:49.806140 2931 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:49.809288 2931 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:49.813287 2931 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:49.828168 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 447705637cdc44668a870a52a7a3871a with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57740
I20260924 00:21:49.829344 1993 raft_consensus.cc:2932] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.856086 2951 tablet_replica.cc:330] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:21:49.859020 2951 raft_consensus.cc:2229] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:49.859825 2951 raft_consensus.cc:2258] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:49.864326 2951 ts_tablet_manager.cc:1905] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:49.918917 2951 ts_tablet_manager.cc:1918] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:49.919425 2951 log.cc:1196] T 447705637cdc44668a870a52a7a3871a P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/447705637cdc44668a870a52a7a3871a
I20260924 00:21:49.921489 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 447705637cdc44668a870a52a7a3871a (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:49.929746 2949 raft_consensus.cc:1050] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:49.932737 2949 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:49.933956 2946 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.941664 2647 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.956595 2143 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:49.964461 1993 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.967042 2949 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:49.969841 2649 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.971294 2063 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:49.983405 1893 catalog_manager.cc:5495] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:49.985354 2949 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:49.987813 2942 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:49.980473 2063 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.027907 1993 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:50.048084 2648 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:50.052464 2145 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:50.054375 2905 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:50.069748 2873 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.057323 2905 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260924 00:21:50.080065 2946 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.083142 2065 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.094972 2648 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.102110 2647 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.108500 1893 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:50.108493 1993 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.119522 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 92bd85e8dadb4876827acd9ff4be377f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:50.123973 1890 catalog_manager.cc:5495] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:50.135316 2141 raft_consensus.cc:2932] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.157104 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bd85e8dadb4876827acd9ff4be377f with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47762
I20260924 00:21:50.195648 2959 tablet_replica.cc:330] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:21:50.196749 2959 raft_consensus.cc:2229] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:50.197507 2959 raft_consensus.cc:2258] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:50.203413 2959 ts_tablet_manager.cc:1905] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:50.206059 2938 raft_consensus.cc:1050] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:50.208767 2938 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:50.203141 2139 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:50.226223 2647 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:50.227979 2957 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:50.228794 2647 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:50.232409 2939 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:50.238976 2957 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.240737 2139 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:50.242646 2935 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:50.248685 2065 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:50.249957 1993 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:50.250379 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2f75d53c3ccf437cac00ba5d83b2c165 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:50.251291 2959 ts_tablet_manager.cc:1918] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:50.251708 2959 log.cc:1196] T 92bd85e8dadb4876827acd9ff4be377f P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/92bd85e8dadb4876827acd9ff4be377f
I20260924 00:21:50.252206 2939 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:50.251379 2957 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260924 00:21:50.258479 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet 92bd85e8dadb4876827acd9ff4be377f (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:50.259554 2065 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.270750 2647 raft_consensus.cc:2932] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.281347 1892 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:50.289186 2933 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.306203 2143 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.315958 1993 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.345003 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f75d53c3ccf437cac00ba5d83b2c165 with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260924 00:21:50.345965 1891 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:50.338824 2647 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.360153 2964 tablet_replica.cc:330] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:21:50.368791 2964 raft_consensus.cc:2229] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:50.369536 2964 raft_consensus.cc:2258] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:50.373107 2063 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:50.385205 2964 ts_tablet_manager.cc:1905] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:50.387466 2143 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:50.391317 2649 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:50.389787 2939 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260924 00:21:50.402204 2933 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:50.412942 2938 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.418848 2964 ts_tablet_manager.cc:1918] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:50.419354 2964 log.cc:1196] T 2f75d53c3ccf437cac00ba5d83b2c165 P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/2f75d53c3ccf437cac00ba5d83b2c165
I20260924 00:21:50.424193 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet 2f75d53c3ccf437cac00ba5d83b2c165 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:50.426685 2143 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.450354 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9ff940e15e1849efb194c908520c212f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:50.428357 2649 raft_consensus.cc:2932] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.472113 1892 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 3 to 4, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:50.484858 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff940e15e1849efb194c908520c212f with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55036
I20260924 00:21:50.486661 2964 tablet_replica.cc:330] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:21:50.487625 2964 raft_consensus.cc:2229] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:50.488224 2964 raft_consensus.cc:2258] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:50.491870 2964 ts_tablet_manager.cc:1905] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:50.513654 2964 ts_tablet_manager.cc:1918] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:50.514093 2964 log.cc:1196] T 9ff940e15e1849efb194c908520c212f P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/9ff940e15e1849efb194c908520c212f
I20260924 00:21:50.515661 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet 9ff940e15e1849efb194c908520c212f (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
W20260924 00:21:51.286010 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 63451.7us
I20260924 00:21:51.567028 1995 consensus_queue.cc:237] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:51.573537 1960 consensus_peers.cc:487] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95ac9e39e8934ecdb84ac79399dcbd9c. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:51.575682 2141 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.576346 2067 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.577741 2905 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:51.578714 2931 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:51.596379 2905 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.598256 2141 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.622318 2067 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.630218 1890 catalog_manager.cc:5495] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.630919 2063 consensus_queue.cc:237] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:51.651587 2035 consensus_peers.cc:487] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 466d562826384d688b157b8830cfd5d1. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:51.653824 2141 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.654024 1993 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.655411 2933 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:51.656729 2935 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:51.705224 2978 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.706923 1993 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.726186 2141 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.742601 1892 catalog_manager.cc:5495] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.779034 2063 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:51.789822 2035 consensus_peers.cc:487] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55201d7dbea64daab55bc0144cc26c13. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:51.794377 1993 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.796983 2935 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:51.799104 2141 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.800487 2978 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:51.806672 2979 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.814180 1995 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.824484 2141 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.848795 2139 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:51.854617 1892 catalog_manager.cc:5495] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:51.862672 2067 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.868300 2873 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260924 00:21:51.872130 2109 consensus_peers.cc:487] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efa7fce6500d4a22b34baf31a7dbb247. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:51.872421 1993 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:51.874754 2957 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:51.902797 2965 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.905267 2063 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.941242 1892 catalog_manager.cc:5495] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.918880 1995 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:21:51.972111 2987 ts_tablet_manager.cc:927] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:51.986326 2987 tablet_copy_client.cc:316] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
I20260924 00:21:51.988019 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet 95ac9e39e8934ecdb84ac79399dcbd9c from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:21:51.988534 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet 95ac9e39e8934ecdb84ac79399dcbd9c from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-95ac9e39e8934ecdb84ac79399dcbd9c
I20260924 00:21:51.997757 2006 tablet_copy_source_session.cc:205] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:52.000438 2987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ac9e39e8934ecdb84ac79399dcbd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:52.027230 2987 tablet_copy_client.cc:791] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:52.032970 2987 tablet_copy_client.cc:655] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:52.037005 2987 tablet_copy_client.cc:531] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:52.046548 2987 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:52.074889 2987 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:52.075902 2987 tablet_bootstrap.cc:492] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:52.076519 2987 ts_tablet_manager.cc:1397] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.001s
I20260924 00:21:52.079162 2987 raft_consensus.cc:348] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.080224 2987 raft_consensus.cc:729] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:52.081293 2987 consensus_queue.cc:260] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.087221 2987 ts_tablet_manager.cc:1428] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20260924 00:21:52.092662 2006 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-95ac9e39e8934ecdb84ac79399dcbd9c received from {username='slave'} at 127.0.0.1:42938
I20260924 00:21:52.093077 2006 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-95ac9e39e8934ecdb84ac79399dcbd9c on tablet 95ac9e39e8934ecdb84ac79399dcbd9c with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:52.179836 2987 ts_tablet_manager.cc:927] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:52.185623 2987 tablet_copy_client.cc:316] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:52.187326 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 55201d7dbea64daab55bc0144cc26c13 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:52.187928 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 55201d7dbea64daab55bc0144cc26c13 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-55201d7dbea64daab55bc0144cc26c13
I20260924 00:21:52.194559 2081 tablet_copy_source_session.cc:205] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:52.197708 2987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55201d7dbea64daab55bc0144cc26c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:52.218611 2987 tablet_copy_client.cc:791] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:52.219197 2987 tablet_copy_client.cc:655] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:52.228855 2987 tablet_copy_client.cc:531] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:52.238869 2987 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:52.263893 2993 ts_tablet_manager.cc:927] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:52.267926 2987 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:52.269177 2987 tablet_bootstrap.cc:492] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:52.269955 2987 ts_tablet_manager.cc:1397] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.015s
I20260924 00:21:52.273116 2987 raft_consensus.cc:348] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.274312 2987 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:52.275081 2987 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.280148 2995 ts_tablet_manager.cc:927] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:52.289326 2987 ts_tablet_manager.cc:1428] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260924 00:21:52.298946 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-55201d7dbea64daab55bc0144cc26c13 received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:52.299387 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-55201d7dbea64daab55bc0144cc26c13 on tablet 55201d7dbea64daab55bc0144cc26c13 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:52.306015 2995 tablet_copy_client.cc:316] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:21:52.308835 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 466d562826384d688b157b8830cfd5d1 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:21:52.310704 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 466d562826384d688b157b8830cfd5d1 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-466d562826384d688b157b8830cfd5d1
I20260924 00:21:52.362255 2993 tablet_copy_client.cc:316] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:21:52.363938 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet efa7fce6500d4a22b34baf31a7dbb247 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:21:52.364471 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet efa7fce6500d4a22b34baf31a7dbb247 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-efa7fce6500d4a22b34baf31a7dbb247
I20260924 00:21:52.372135 2155 tablet_copy_source_session.cc:205] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:52.375540 2993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa7fce6500d4a22b34baf31a7dbb247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:52.384244 2081 tablet_copy_source_session.cc:205] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:21:52.387684 2995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 466d562826384d688b157b8830cfd5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:21:52.391635 2993 tablet_copy_client.cc:791] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:52.392395 2993 tablet_copy_client.cc:655] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:52.413357 2993 tablet_copy_client.cc:531] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:52.416095 2995 tablet_copy_client.cc:791] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:21:52.416814 2995 tablet_copy_client.cc:655] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:21:52.422655 2995 tablet_copy_client.cc:531] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:21:52.424043 2993 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:52.465605 2995 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:21:52.491568 2993 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:52.492588 2993 tablet_bootstrap.cc:492] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:52.493320 2993 ts_tablet_manager.cc:1397] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.070s user 0.031s sys 0.032s
I20260924 00:21:52.496934 2993 raft_consensus.cc:348] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.497895 2993 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:52.498598 2993 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.501802 2993 ts_tablet_manager.cc:1428] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260924 00:21:52.504341 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-efa7fce6500d4a22b34baf31a7dbb247 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:21:52.504757 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-efa7fce6500d4a22b34baf31a7dbb247 on tablet efa7fce6500d4a22b34baf31a7dbb247 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:52.530980 2995 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:21:52.532058 2995 tablet_bootstrap.cc:492] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:21:52.533429 2995 ts_tablet_manager.cc:1397] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.013s
I20260924 00:21:52.536337 2995 raft_consensus.cc:348] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.537424 2995 raft_consensus.cc:729] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:21:52.538130 2995 consensus_queue.cc:260] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:52.541533 2995 ts_tablet_manager.cc:1428] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260924 00:21:52.544013 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-466d562826384d688b157b8830cfd5d1 received from {username='slave'} at 127.0.0.1:43194
I20260924 00:21:52.544384 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-466d562826384d688b157b8830cfd5d1 on tablet 466d562826384d688b157b8830cfd5d1 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:52.683277 2647 raft_consensus.cc:1203] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:52.703485 2649 raft_consensus.cc:1203] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:52.733595 2649 raft_consensus.cc:1203] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:52.763108 2649 raft_consensus.cc:1203] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260924 00:21:53.093519 2970 raft_consensus.cc:1050] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:53.097671 2970 consensus_queue.cc:237] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:53.108691 2649 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.108788 2143 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.111310 2065 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.113000 2970 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.115069 3003 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:53.116151 2969 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.149438 2970 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.184420 2065 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.191186 2979 raft_consensus.cc:1050] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:53.186072 2139 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.193758 2979 consensus_queue.cc:237] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:53.202670 1891 catalog_manager.cc:5495] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.198113 2649 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.206368 2873 raft_consensus.cc:1050] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:53.212375 2873 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:53.237664 2647 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.240641 2141 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.240973 1994 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.243818 2980 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.245911 2935 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:53.258826 1994 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.260548 2648 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.260490 2873 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.262557 2065 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.262827 2980 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:53.264863 3000 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.280849 3000 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.298501 3011 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.333035 2648 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.337124 2999 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.326956 2141 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.340719 2649 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.353801 2063 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.352309 1995 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.367260 1996 consensus_queue.cc:237] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:53.370631 3013 raft_consensus.cc:1050] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:21:53.376515 3013 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:53.380218 2646 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:53.386243 2931 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:53.391027 2647 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.392191 2145 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:53.394429 2979 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.396709 2905 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.399405 1890 catalog_manager.cc:5495] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.405498 3003 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:53.426196 2143 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.433928 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 95ac9e39e8934ecdb84ac79399dcbd9c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:53.436775 2143 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.439445 2649 raft_consensus.cc:2932] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.428267 3013 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:21:53.469815 1891 catalog_manager.cc:5495] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 1 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:53.477377 3011 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.480122 2141 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.488063 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 95ac9e39e8934ecdb84ac79399dcbd9c with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47762
W20260924 00:21:53.489805 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 68220.6us
I20260924 00:21:53.489882 1890 catalog_manager.cc:5495] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.512358 2065 consensus_queue.cc:237] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:53.523923 1994 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260924 00:21:53.520067 3028 tablet_replica.cc:330] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:21:53.534034 3028 raft_consensus.cc:2229] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:53.534581 1994 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.555166 3028 raft_consensus.cc:2258] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:53.555825 2979 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:21:53.564931 2649 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:53.569358 2979 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:21:53.572594 1893 catalog_manager.cc:5495] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:53.567705 2648 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.588531 3011 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.600528 1993 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:53.603422 3028 ts_tablet_manager.cc:1905] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:53.606487 2935 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260924 00:21:53.610756 2648 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.622967 2141 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:53.629500 1996 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:53.631055 2646 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:53.632658 3000 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:53.635233 2999 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:53.637249 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 466d562826384d688b157b8830cfd5d1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:53.632694 1993 raft_consensus.cc:2932] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.662317 1890 catalog_manager.cc:5495] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.667924 2873 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.685048 2646 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.689613 1996 raft_consensus.cc:2932] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.706274 1995 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.718863 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet efa7fce6500d4a22b34baf31a7dbb247 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:53.725296 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 466d562826384d688b157b8830cfd5d1 with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57740
I20260924 00:21:53.738292 1893 catalog_manager.cc:5495] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 3 to 4, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 1 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:53.752936 3028 ts_tablet_manager.cc:1918] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:53.753499 3028 log.cc:1196] T 95ac9e39e8934ecdb84ac79399dcbd9c P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/95ac9e39e8934ecdb84ac79399dcbd9c
I20260924 00:21:53.755399 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet 95ac9e39e8934ecdb84ac79399dcbd9c (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:53.767601 3031 tablet_replica.cc:330] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:21:53.772372 3031 raft_consensus.cc:2229] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:53.773765 3031 raft_consensus.cc:2258] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:53.777468 3031 ts_tablet_manager.cc:1905] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:53.797447 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet efa7fce6500d4a22b34baf31a7dbb247 with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47762
I20260924 00:21:53.798568 3028 tablet_replica.cc:330] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:21:53.799639 3028 raft_consensus.cc:2229] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:53.800426 3028 raft_consensus.cc:2258] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:53.804899 3028 ts_tablet_manager.cc:1905] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:53.825440 3028 ts_tablet_manager.cc:1918] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:53.825960 3028 log.cc:1196] T efa7fce6500d4a22b34baf31a7dbb247 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/efa7fce6500d4a22b34baf31a7dbb247
I20260924 00:21:53.828338 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet efa7fce6500d4a22b34baf31a7dbb247 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:53.838728 3031 ts_tablet_manager.cc:1918] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:53.839336 3031 log.cc:1196] T 466d562826384d688b157b8830cfd5d1 P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/466d562826384d688b157b8830cfd5d1
I20260924 00:21:53.841741 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 466d562826384d688b157b8830cfd5d1 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:54.591495 2063 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:54.597317 1992 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:54.598592 2935 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:54.599462 2647 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260924 00:21:54.602486 2935 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:21:54.606210 2979 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:54.607767 1992 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:54.611287 2647 raft_consensus.cc:2932] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:54.615921 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 55201d7dbea64daab55bc0144cc26c13 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:21:54.624617 1890 catalog_manager.cc:5495] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 3 to 4, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 1 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:21:54.638806 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 55201d7dbea64daab55bc0144cc26c13 with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57740
I20260924 00:21:54.640384 3049 tablet_replica.cc:330] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:21:54.641108 3049 raft_consensus.cc:2229] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:21:54.641757 3049 raft_consensus.cc:2258] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:21:54.643767 3049 ts_tablet_manager.cc:1905] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:21:54.656214 3049 ts_tablet_manager.cc:1918] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260924 00:21:54.656687 3049 log.cc:1196] T 55201d7dbea64daab55bc0144cc26c13 P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/55201d7dbea64daab55bc0144cc26c13
I20260924 00:21:54.658233 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 55201d7dbea64daab55bc0144cc26c13 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:21:55.865566 3052 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:55.867555 3052 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:55.872267 3052 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:55.915791 3040 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:55.916916 3040 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:55.917624 3035 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:55.919953 3035 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:55.922838 3035 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:55.932090 3035 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:55.932978 3035 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:55.940037 3035 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:55.943481 3040 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:55.971309 1925 debug-util.cc:398] Leaking SignalData structure 0x7b080003ee80 after lost signal to thread 815
W20260924 00:21:56.000363 1925 debug-util.cc:398] Leaking SignalData structure 0x7b080008ed20 after lost signal to thread 1928
I20260924 00:21:56.002591 3040 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.009054 3055 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.003445 3035 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.014942 3053 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.016959 3053 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.017452 3064 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.017165 3040 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.018149 3064 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.019438 3053 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.009938 3055 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.020262 3060 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.020608 3040 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.022338 3064 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.022970 3064 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.022338 3055 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.023578 3064 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.005730 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.026580 3045 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.027403 3045 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:21:56.033147 2166 debug-util.cc:398] Leaking SignalData structure 0x7b08001ff4a0 after lost signal to thread 815
W20260924 00:21:56.033512 1925 debug-util.cc:398] Leaking SignalData structure 0x7b0800165260 after lost signal to thread 2020
I20260924 00:21:56.021060 3060 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.073336 3064 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.029245 3062 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.013517 3035 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:56.093384 3036 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.094290 3036 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.096877 3052 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.097046 3036 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.097256 3053 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.098534 3052 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.092394 3055 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:21:56.101351 1925 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc900 after lost signal to thread 2095
I20260924 00:21:56.150034 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.099834 3064 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.150496 3036 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:21:56.161453 2166 debug-util.cc:398] Leaking SignalData structure 0x7b08001b28a0 after lost signal to thread 1928
I20260924 00:21:56.161564 3036 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:56.163218 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800327ba0 after lost signal to thread 2020
I20260924 00:21:56.163966 3036 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.157290 2969 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:21:56.164728 2166 debug-util.cc:398] Leaking SignalData structure 0x7b080033ad40 after lost signal to thread 2095
I20260924 00:21:56.166625 3068 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.167397 3035 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.167181 3068 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.164994 2969 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.122439 2905 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.121763 3054 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.122432 3003 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.096861 3065 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.112368 2970 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.121870 3039 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.179330 3040 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.180811 3035 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.203019 3066 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.206956 3067 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.206682 3066 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.207626 3067 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.214068 3059 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:21:56.207989 1925 debug-util.cc:398] Leaking SignalData structure 0x7b0800283a60 after lost signal to thread 2169
I20260924 00:21:56.184307 3055 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.206665 3068 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.211266 3040 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.212966 2905 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.103065 3062 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.202419 3064 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.217609 2969 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.226928 3037 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.216506 3045 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.205699 3053 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.237097 3056 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.214105 3035 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.225095 3068 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.245879 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.238024 3056 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.222951 3060 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.252285 3040 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.252229 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.258380 2905 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.259133 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.259296 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.259443 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.261353 3063 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.231575 2970 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.214735 3059 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.262239 3063 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.269652 2970 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:56.265726 1925 debug-util.cc:398] Leaking SignalData structure 0x7b08002ca400 after lost signal to thread 2673
I20260924 00:21:56.256886 3035 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.270358 3069 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.287268 3045 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.280939 3052 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.254649 3003 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.253067 3039 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:56.288431 2166 debug-util.cc:398] Leaking SignalData structure 0x7b08002d7080 after lost signal to thread 2169
I20260924 00:21:56.238377 3037 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.224134 3065 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.297358 3045 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.297933 3058 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.282349 3067 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.303900 3035 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.288625 3051 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.304466 3036 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.280395 3068 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.242944 3054 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.315424 3036 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.316151 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.316946 1995 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:21:56.317229 3045 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.317737 3036 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.317852 3045 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.318331 2905 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.318558 3045 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.318929 2905 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.301643 3069 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:56.330591 3055 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.330863 2905 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.325528 3040 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.331629 3040 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.313939 3067 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.335156 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.335098 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.336773 2614 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.337567 3054 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.337839 3040 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.325730 2970 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.346207 3067 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.346882 2970 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.325939 3036 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.336064 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.338961 3064 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.339905 2649 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.348672 3067 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.335441 3060 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.345646 3056 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.341586 3077 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.359956 3003 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.340718 3066 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.356730 3064 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.366098 2905 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.359350 3053 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.339094 3058 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.384922 3077 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.389463 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.368836 3067 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.369026 3055 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.374830 3059 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.376549 3037 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.395063 3077 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.394558 3055 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.395668 3078 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.377918 3062 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.376573 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.356071 3074 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:21:56.381310 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800265940 after lost signal to thread 2673
I20260924 00:21:56.371764 3064 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.381910 3045 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.382395 3068 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.382891 3075 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.360847 3054 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.383572 2970 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.383740 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.384461 3036 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.341114 3035 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.391299 2069 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.391218 3067 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.382573 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.394619 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.378829 3060 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.396852 2905 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.401618 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.410933 3053 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.367403 2969 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.414522 3037 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.397258 3040 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.344203 3051 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.418360 3076 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:21:56.369046 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 80792us
I20260924 00:21:56.409916 3064 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.407284 3055 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.419000 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.427980 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.380025 3039 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.409055 3003 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.414762 3082 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.436270 2649 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.437093 3070 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.416388 3066 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.441390 3068 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.442768 3079 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.442690 3066 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.443198 3068 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.396231 3078 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.437418 2970 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.444024 3060 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.453615 3071 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.454046 3066 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.454407 3068 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.454133 3037 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.453689 2969 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.462827 1995 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:21:56.464972 3035 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.465360 3058 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.465076 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.465795 2065 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.472627 3083 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.472561 2970 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.446228 3065 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.476058 2905 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.445873 3039 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.414755 3075 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.470122 2068 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:56.443504 3079 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.466285 3003 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.486353 3068 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.487399 2969 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.487030 3040 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.479432 3035 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.460745 3053 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.462896 3055 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.445681 2071 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.441558 3054 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.473477 3083 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.516125 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.458695 3080 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.483752 3060 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.511147 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.507295 3069 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:56.430467 3074 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.516288 3052 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.539538 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.539242 3067 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.532804 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.455245 3070 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.533246 3059 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.480587 3071 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.518270 3056 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.525650 3062 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.518814 3065 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.543516 3059 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.470114 3076 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.565449 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.565987 3070 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.566958 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.568078 3077 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.568666 3077 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.528597 3039 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.527093 3055 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.493996 3082 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.590489 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.563158 3036 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.536717 3068 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.546689 3060 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.601413 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.599789 2905 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.560431 3035 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.602061 1991 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:21:56.573702 3056 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.605814 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.580693 3062 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.580566 3052 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.598955 3037 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.611809 2614 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.608621 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.616848 3054 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.613663 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.614914 3035 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.614626 2905 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.611883 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.613080 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.601526 2969 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.604645 3051 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.617834 2141 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:21:56.621058 2143 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:56.625937 3066 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.626222 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.626362 3066 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.615708 2970 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.620048 3077 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.626876 2139 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:21:56.606624 3065 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.631314 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.624482 3079 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.630481 3035 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.635125 2905 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.633105 3075 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.636425 3085 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.638010 3085 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.638103 3068 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.638401 3085 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.638443 3082 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.632467 3040 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.628365 3066 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.641165 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.624440 3063 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.629823 3003 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.627164 3054 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.638856 3035 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.643250 3053 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.649845 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.650200 3035 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.629966 3076 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.650161 3055 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.628031 3087 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.627903 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.631551 3083 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.590305 3080 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.654244 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.646211 2614 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.632195 2969 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.658998 3088 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.659756 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.660249 3065 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.660693 3078 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.661993 2066 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.640867 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.657050 2144 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:21:56.665890 2613 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:56.654497 3074 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.666595 3059 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.668340 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.674163 3088 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.663050 3054 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.674727 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.674968 3083 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.694916 3083 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.696926 3083 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.675213 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.674991 3064 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.697450 3088 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.676152 3060 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.678297 3068 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.654199 3087 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.683017 3052 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.683662 3056 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.683779 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.716483 2068 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.679144 3039 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.683935 3062 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.722215 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.722357 3087 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.684551 3071 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.689253 3079 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.724059 3079 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.711223 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.726119 3079 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.681056 3076 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.726323 2969 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.726917 3076 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.728742 3076 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.723392 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.730180 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.699615 2613 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.703030 3077 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.731346 3045 raft_consensus.cc:2781] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.731277 3077 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.732326 2613 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:56.703375 2643 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:56.703454 3068 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.740516 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.740904 3068 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.742005 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.742648 2143 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:56.705818 3084 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:56.745339 3084 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.711162 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.745757 3084 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:56.710310 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.711762 3080 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.712224 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.712664 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.712329 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.713557 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.731742 3045 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.705641 3064 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.725119 2070 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.737406 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.738897 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.769402 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.745183 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.697116 3054 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.711544 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.772903 2065 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.774981 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.776266 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.746584 2063 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.754451 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.755486 2069 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.757336 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.756011 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.770494 2066 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.772231 3077 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.783949 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.756325 3076 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.757854 3045 raft_consensus.cc:3037] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.786818 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.757349 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.782362 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.779534 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.786513 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.758025 2062 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:56.789724 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.790375 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.788592 2140 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:56.779876 3064 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.790598 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.794605 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.783420 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.807590 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.805953 3064 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.812969 3054 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.813956 3054 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.815999 3054 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.816623 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.817577 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.817764 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.817914 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.818997 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.813503 3064 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.820497 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.820719 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.821218 1996 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:56.824147 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.826905 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.828316 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.832289 3039 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.810046 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.810439 3085 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.835001 2137 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:56.832762 3039 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.839051 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.839253 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.839989 3039 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.791364 3066 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.837782 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:56.803690 3076 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.847312 3076 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.839068 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.849926 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.850594 1993 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:56.853979 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.856036 3064 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.856588 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.857204 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.857825 1993 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:56.858572 3064 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.861688 2035 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.861254 3068 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.862381 3079 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.862739 3079 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.863085 3079 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.858632 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.864727 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.869938 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.874090 3068 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.869966 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.876952 2969 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.877364 2969 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.877667 2969 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.873512 3085 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.878703 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.880633 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.881665 2035 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.883822 3087 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:56.884083 3074 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.884269 3087 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.884568 3074 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.885521 3087 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:56.885676 3066 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.885708 3074 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.883046 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.873063 3084 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.886390 3076 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:56.888267 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.889987 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.891013 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.891408 3051 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.891659 2905 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.891837 3051 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.892048 2905 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.892298 3051 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.892441 2905 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.892968 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.895715 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.896924 3035 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.896924 3090 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.897574 3035 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.897967 3090 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.898208 3035 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.873063 3088 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:56.898377 3090 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.901652 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:56.904211 1995 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:56.904950 1960 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.907220 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:56.907784 1995 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.908103 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:56.908717 1996 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.901794 3076 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.914289 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.914886 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.917678 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:56.917816 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.933844 1996 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:56.920588 2969 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.921767 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:56.940650 3077 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.941044 3077 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:56.941479 3077 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.931926 1993 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.934882 3040 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.942783 3040 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.943195 3040 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.922634 2905 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.937582 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.946707 2905 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.934437 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.939657 2614 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.942152 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.936743 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.913357 3085 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.949004 2905 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260924 00:21:56.949894 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 217638us
I20260924 00:21:56.956596 2905 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.948696 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:56.951984 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.952634 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.953642 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.954200 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.957309 2905 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.934119 3076 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.957410 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.957569 1993 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:56.962345 1995 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:56.963375 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:56.964164 3068 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.964412 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:56.964221 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.966074 3089 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:56.966137 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.966516 3089 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.966965 3089 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.966336 3003 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:56.970948 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:56.963728 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.957649 3085 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:56.964632 3068 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:56.965648 3076 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.964521 2614 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:56.971050 3075 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:56.961787 3066 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:56.975427 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:56.941044 3087 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:56.977648 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:56.978513 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.973217 3088 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:56.975742 3068 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.974364 3085 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:56.989940 3057 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:56.981830 2613 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:56.974792 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.994017 3084 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.994647 3051 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:56.995030 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:56.988509 3064 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:56.986366 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:56.999338 3087 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:56.988914 3066 raft_consensus.cc:686] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:57.006925 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:56.987761 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.008460 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.009531 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.002979 2614 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:57.004522 3064 raft_consensus.cc:686] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:57.012163 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.012856 2064 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:21:57.012416 3064 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.014437 3039 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.018157 3051 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.021634 3039 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.024343 2969 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.025403 3074 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.025926 3068 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.027621 3074 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.027765 3068 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.028764 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.031271 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.032017 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.032418 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.006448 3093 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.030699 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.037110 3085 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.039187 3085 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.040316 3090 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.042150 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.042330 3090 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:56.996510 3056 consensus_queue.cc:1046] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260924 00:21:57.046231 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.049824 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.034049 2905 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.051334 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.052897 2905 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.010747 3083 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.055505 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.056171 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.056856 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.044981 3063 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.057698 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.060899 3066 consensus_queue.cc:237] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.001720 3040 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.061658 3079 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.063067 3045 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:21:57.054939 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 61872.4us
I20260924 00:21:57.065068 3072 raft_consensus.cc:3032] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.074543 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.081049 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.083168 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.083909 1996 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:57.085402 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.079161 3089 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.087325 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.088860 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.071698 3068 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:57.089989 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.091383 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.091274 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.091706 3089 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.092344 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.092453 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.093575 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.093984 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.110816 1991 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:57.112051 3035 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.113615 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.114091 3035 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.082525 3072 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.088034 1992 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:57.096238 3079 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.096278 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.098109 3045 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.098573 2642 raft_consensus.cc:1226] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.117951 3072 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.121182 3072 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.088989 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.071740 3077 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.106284 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.106691 2614 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.107903 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.109663 1994 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:57.116022 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.114675 1996 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:57.140163 1990 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:57.140394 3093 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:21:57.128221 3063 consensus_queue.cc:1046] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:21:57.140340 1993 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:57.141359 3063 raft_consensus.cc:3032] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.141739 3063 raft_consensus.cc:729] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.142802 3063 consensus_queue.cc:260] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.144599 3063 raft_consensus.cc:3037] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.130259 3040 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.088776 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.136094 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.159217 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.137580 3076 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.104332 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.122491 3068 raft_consensus.cc:686] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:57.160018 3077 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:21:57.187141 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 216716us
I20260924 00:21:57.192227 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.192794 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.193472 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.194010 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.194451 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.195151 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.195472 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.195700 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.196256 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.196535 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.196717 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.198868 2642 raft_consensus.cc:1226] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.198931 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.199841 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.203764 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.205174 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.194375 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.211501 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.212311 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.219046 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:21:57.219067 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.221330 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:57.221837 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.218689 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.223217 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.224437 2066 raft_consensus.cc:1226] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.225505 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.227566 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.228407 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.228833 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.225512 2064 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.230053 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:21:57.230427 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.232134 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.233664 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.235530 2069 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.237289 3077 consensus_queue.cc:1046] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:21:57.238171 3077 consensus_queue.cc:1225] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer 9e79063532384ada8ea8f8205478adcf log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260924 00:21:57.238930 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.239045 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.238106 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.240453 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.241657 3076 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.241595 2071 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:57.242219 2642 raft_consensus.cc:1226] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3. Ops: []
I20260924 00:21:57.242820 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.243010 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.201882 3068 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.243304 2613 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.243839 2646 raft_consensus.cc:1226] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.247792 2613 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.251732 3066 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.252342 3066 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.253882 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.254495 3037 raft_consensus.cc:3032] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.254890 3037 raft_consensus.cc:729] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.255072 2613 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.255739 3037 consensus_queue.cc:260] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.256567 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.258718 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.262581 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.263120 3064 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.263279 3093 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.263139 3066 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.264185 3066 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:57.264149 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.264704 3066 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.267572 3093 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.257088 3037 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.268913 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.244660 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.245342 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.243984 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:57.271545 3076 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.270254 1892 catalog_manager.cc:5495] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:57.275950 3066 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.278849 2064 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:57.275817 1892 catalog_manager.cc:5495] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260924 00:21:57.285494 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 64309.3us
I20260924 00:21:57.289244 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.290006 2613 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.294458 3003 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.294461 3093 raft_consensus.cc:2781] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:57.295269 3093 raft_consensus.cc:686] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:57.296334 3093 consensus_queue.cc:237] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.303365 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:21:57.303799 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 80587.7us
I20260924 00:21:57.304432 3003 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.305824 2646 raft_consensus.cc:1226] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.307488 3079 consensus_queue.cc:1046] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:21:57.308573 3079 raft_consensus.cc:3032] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.308980 3079 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.310048 3079 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.311479 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.312875 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.311076 3066 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:57.314538 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.314451 3003 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.316869 3040 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.316972 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.316210 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.316341 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.320672 3040 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.318302 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.321528 2646 raft_consensus.cc:1226] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.322284 3079 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.323117 3037 consensus_queue.cc:1046] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:21:57.324211 3037 consensus_queue.cc:1225] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer 9e79063532384ada8ea8f8205478adcf log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260924 00:21:57.324672 3063 raft_consensus.cc:3032] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.325053 3063 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.325381 3040 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.326843 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.313210 3098 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.326267 3063 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.329653 3063 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.322148 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.338765 3040 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.339234 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.341722 3040 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.328617 3066 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.343664 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.344774 3040 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.345468 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.346490 3040 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:21:57.348399 1993 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.348896 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.349668 3100 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.353243 2109 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.354010 3100 raft_consensus.cc:2781] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.354341 3100 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.354661 3100 raft_consensus.cc:3037] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.356196 2110 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.356968 3098 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.358987 2110 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.360703 3104 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.363317 2110 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.364055 3098 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.367388 3100 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.366953 2110 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.369669 3100 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:57.370198 3100 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.374315 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.375131 2064 raft_consensus.cc:2379] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:21:57.375917 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:21:57.378198 2109 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.378655 2110 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.379402 3054 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.379961 3100 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.381529 2110 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.382329 3100 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.382758 3100 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.383157 3100 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.387282 2110 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.388401 3103 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.388501 2110 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.390483 2110 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.390663 3100 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.401331 3054 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.401734 3100 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.403631 2110 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.403565 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.413167 3100 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.413595 2110 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.414546 3054 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.413708 1892 catalog_manager.cc:5495] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:57.405244 1989 raft_consensus.cc:1226] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.415763 2110 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.407301 1994 raft_consensus.cc:1226] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.416554 3054 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.413594 3100 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.417214 3100 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.417693 2110 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.422036 3054 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.427498 2110 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.397341 3103 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:57.428406 3103 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.429734 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.430980 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.440258 2141 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.440521 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.440557 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.441604 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.442111 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.442582 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.443315 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.443610 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.431329 1992 raft_consensus.cc:1226] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.431880 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.432966 3104 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.434446 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.444654 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.435261 2110 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.435386 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.432919 1987 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.462075 2110 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.437336 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.437969 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.463541 3098 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.463991 3098 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.464337 3098 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.447079 3077 consensus_queue.cc:1046] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:57.465875 3077 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.466291 3077 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.467272 3077 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.452497 2065 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.434625 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.447165 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.470613 3093 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:21:57.471704 3104 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:21:57.444582 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:57.468858 3077 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.463244 2110 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.482594 3104 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.454954 3103 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:57.482990 3104 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.483417 3104 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.423008 1892 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:57.489729 2110 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.427337 3100 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.491411 2110 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.492574 2110 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.493902 2110 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.495395 2110 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.490950 3054 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.495961 3080 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.496773 3062 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.497078 3097 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.500015 3076 consensus_queue.cc:1046] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260924 00:21:57.506916 3036 raft_consensus.cc:3032] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:57.507239 3036 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:57.508074 3036 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.502228 3092 consensus_queue.cc:1046] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260924 00:21:57.511166 2141 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.512856 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.502740 3054 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.503818 3056 consensus_queue.cc:1046] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260924 00:21:57.511677 3036 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:57.504318 3080 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.517719 1960 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.515960 3040 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.505206 3097 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.504285 3062 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.521221 3097 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.521484 3062 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.517206 3095 consensus_queue.cc:1046] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.523317 3040 raft_consensus.cc:3032] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.523680 3040 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:57.519011 3056 raft_consensus.cc:3032] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.519673 3003 raft_consensus.cc:3032] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.512508 3076 consensus_queue.cc:1046] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260924 00:21:57.515228 3054 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.520114 3061 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.527956 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:57.524471 3040 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.520723 3080 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.531035 2036 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.525679 3092 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.527269 3003 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:57.526335 3056 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.530192 3054 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.533241 3040 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.539669 3035 raft_consensus.cc:3032] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.540094 3035 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:57.542223 2036 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.533180 3085 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.543175 3070 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.541384 3100 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.546027 3090 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.545861 3062 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.547133 3071 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.547483 3071 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.548290 3071 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.548324 2036 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.528383 3095 consensus_queue.cc:1046] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260924 00:21:57.556387 3056 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.557964 3056 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.557587 3003 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.547951 3062 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.558786 3003 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.555365 3090 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.540913 3035 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.561913 3040 raft_consensus.cc:3032] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.577584 3040 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:57.572046 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.575719 3035 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.564091 3071 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:21:57.587029 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 53103.7us
I20260924 00:21:57.596843 2071 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.595726 3080 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.600780 2036 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:21:57.603670 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 108405us
I20260924 00:21:57.605780 2036 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.604790 2110 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.597769 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.608153 2036 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.650769 2036 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.657390 2036 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.662256 3072 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.663777 1993 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.668524 3080 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.668382 3045 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:21:57.659186 3078 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.654640 3079 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.600023 3098 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.607853 3083 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.608965 3056 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.678620 3100 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.683177 3100 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.683537 3100 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.659554 3040 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.684891 3040 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.603636 3104 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.690793 2143 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.691077 3104 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.604684 3097 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.694288 1990 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.694577 3097 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.695353 2141 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.696426 3098 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:21:57.687822 2905 consensus_queue.cc:1159] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "948356bf66a941ef8249ecbd0ec94f6d" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260924 00:21:57.707157 3103 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.675015 3054 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:57.718573 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 113843us
I20260924 00:21:57.720126 2110 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.723481 3054 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.724074 3054 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:57.724397 3054 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:57.724470 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.724768 3054 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.725749 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.726603 3103 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.726747 3035 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.727133 3103 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.727769 2137 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3. Ops: []
I20260924 00:21:57.729645 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.731021 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.732285 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.734938 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.713029 3068 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:21:57.738828 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.740253 1996 raft_consensus.cc:1226] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.741031 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.741513 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.745168 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.745424 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.746042 1993 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.746421 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.746881 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.747347 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.748478 1995 raft_consensus.cc:1226] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.749559 1995 raft_consensus.cc:1226] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:21:57.750816 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:21:57.752738 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 95717us
I20260924 00:21:57.753851 1994 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.753652 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.754230 1992 raft_consensus.cc:1226] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.755605 2036 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.737860 3109 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.734771 3076 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.737736 3003 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.734997 3035 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.762343 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.761301 2065 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:57.762944 2036 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.756757 3056 consensus_queue.cc:1046] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260924 00:21:57.756574 1987 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.756374 3078 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.756585 3070 consensus_queue.cc:1046] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260924 00:21:57.770200 2143 raft_consensus.cc:1226] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.766611 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.758747 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.774811 3090 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:21:57.775878 3070 raft_consensus.cc:3032] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.776341 3070 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.775089 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.762920 3100 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.778045 3056 raft_consensus.cc:3032] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.778455 3056 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.778317 3078 consensus_queue.cc:1046] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:57.781873 3079 consensus_queue.cc:1046] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:57.781975 3056 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.783285 3056 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.784821 3100 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.783159 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.770073 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.785022 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:57.768937 1990 raft_consensus.cc:1226] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.777181 3070 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.792968 3070 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.777042 3054 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.807598 3079 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:57.808960 3079 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.805399 3054 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:57.809783 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:57.832036 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.833266 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.836592 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.837555 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.837885 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.806217 3083 raft_consensus.cc:3032] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.838734 3083 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.839459 3083 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.840515 3083 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.837495 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.809849 3079 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.846347 3079 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:57.850116 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:57.850951 2140 consensus_queue.cc:237] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:57.852162 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.853930 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:57.858295 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.859791 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.856371 3054 consensus_queue.cc:1046] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.856808 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:21:57.862160 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 87154.1us
I20260924 00:21:57.862885 2036 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.862905 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.863749 3040 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.864415 3040 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.866072 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.867972 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.860781 3104 consensus_queue.cc:1046] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260924 00:21:57.869763 3109 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.870041 3078 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.869884 3040 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.869585 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.865666 3118 consensus_queue.cc:1046] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260924 00:21:57.862069 3003 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.870041 1989 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.879151 2110 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.879904 3080 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.881227 2137 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:57.882134 3103 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.882581 3103 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.883190 3078 consensus_queue.cc:1046] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.883386 3035 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:21:57.872555 3054 raft_consensus.cc:3032] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.886096 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.882069 2036 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.888144 2036 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:57.879384 3115 consensus_queue.cc:1046] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.888940 3003 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.882699 3080 consensus_queue.cc:1046] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.889555 3115 raft_consensus.cc:3032] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.889919 3115 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.890082 3080 raft_consensus.cc:3032] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.890450 3080 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.890771 3115 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.891202 3080 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.892252 3115 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.868498 3090 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.892879 3117 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.876645 3100 consensus_queue.cc:1046] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.895121 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.896276 3117 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.872905 3063 consensus_queue.cc:1046] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:21:57.897272 3100 raft_consensus.cc:3032] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:21:57.897671 3100 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.898057 3063 raft_consensus.cc:3032] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.898531 3063 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.898483 3100 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:57.899899 3100 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:57.899701 3063 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:57.901871 3061 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.901016 3063 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.902256 3061 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:57.903190 3061 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.904369 3061 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.905911 1989 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.887122 3054 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.907714 3054 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.908931 3054 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.909637 3104 raft_consensus.cc:3032] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.910007 3104 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.910717 3104 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.911933 3035 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.912031 3104 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.878415 3110 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.878661 3114 consensus_queue.cc:1046] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:57.913802 3118 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.914145 3118 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.914366 3114 raft_consensus.cc:3032] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.913076 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.914722 3114 raft_consensus.cc:729] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.915110 3035 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.915035 3118 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.915537 3114 consensus_queue.cc:260] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.916291 3118 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.916492 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.892575 3080 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.919246 3035 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.883476 3103 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.919288 3110 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:57.921489 3110 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:57.921840 3110 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:57.920462 3103 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.916668 3114 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.924038 3110 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:57.922634 3068 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260924 00:21:57.927279 2137 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.933800 2109 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.918571 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.932816 3119 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:21:57.935577 3035 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.937490 1960 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.941593 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.942497 3035 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.943114 3003 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.945518 2065 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.947307 2065 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.948691 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.951066 2069 raft_consensus.cc:1261] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:57.955063 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.962023 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.963263 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.964143 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.964396 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.969866 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.946955 3110 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:57.967710 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:57.977618 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:57.979640 3035 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:57.997783 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:21:58.011260 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 77572.1us
I20260924 00:21:58.021565 3054 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:58.023005 2067 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:21:58.030959 2062 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:58.028656 2070 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:58.017262 3040 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:58.003360 3119 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260924 00:21:58.003533 3045 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260924 00:21:58.008706 3068 consensus_queue.cc:1035] T 7fa219f0283c4632b6601e60980ae25a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260924 00:21:58.066115 1994 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.088532 2065 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:58.071979 3103 consensus_queue.cc:1046] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260924 00:21:58.112154 1994 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.121927 1994 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.127600 3097 raft_consensus.cc:3032] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.128052 3097 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.129232 3097 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:21:58.130398 2109 consensus_peers.cc:487] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2ef9fa9e5dd48218a5c243de0c0e49a. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:21:58.134624 2140 raft_consensus.cc:1226] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.134908 3126 consensus_queue.cc:1046] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:58.131996 3097 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.138672 3118 raft_consensus.cc:3032] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.139122 3118 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.140071 3118 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.123790 3110 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260924 00:21:58.145758 3118 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.146082 3103 raft_consensus.cc:3032] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.146543 3103 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.147392 3103 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.148425 3103 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.161019 2140 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.168357 2109 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.172173 2140 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.174863 3064 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:58.177214 2142 raft_consensus.cc:1226] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.176819 2109 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:58.177436 3064 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.178365 3127 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:58.172780 3110 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:58.179858 2109 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:58.183391 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:58.190688 2109 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:58.190457 3104 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:58.195412 3064 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:58.181515 3126 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:58.196498 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:58.197149 3127 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:21:58.197882 1990 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:58.184401 3110 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.200528 3110 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.202559 2109 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.212713 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:58.221395 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.220854 2065 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.230633 2613 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.231760 3064 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:58.232110 3064 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:58.232499 3064 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.230530 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.235558 2109 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.236358 3126 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:58.243610 3126 raft_consensus.cc:686] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.244047 2067 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:21:58.236367 3127 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:58.248493 3127 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.248718 3104 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:58.249361 3104 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.249518 3127 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.250569 3104 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.244481 3126 consensus_queue.cc:237] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.256814 3103 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:58.287580 3103 raft_consensus.cc:686] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.318120 3064 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.323431 3103 consensus_queue.cc:237] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.333101 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:58.331779 3064 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:58.333341 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:58.355525 2643 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:21:58.359107 3061 consensus_queue.cc:1046] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:58.359578 3063 consensus_queue.cc:1046] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:58.360810 3063 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:58.362080 3090 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.362545 3090 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:58.362028 3117 raft_consensus.cc:3032] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.362950 1989 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:58.363219 3117 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:58.363384 3090 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.364063 3117 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.364681 3090 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.365612 3061 raft_consensus.cc:3032] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.360862 3079 consensus_queue.cc:1046] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:58.366025 3117 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.366799 3078 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.367151 3078 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:58.367923 3078 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.369438 2069 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.365981 3061 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:58.372066 2613 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:58.370939 3061 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.398480 3064 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:58.399469 3064 raft_consensus.cc:686] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:58.401461 3078 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.402335 3064 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.413291 3126 consensus_queue.cc:1046] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260924 00:21:58.423033 3126 raft_consensus.cc:3032] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.423454 3126 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.423861 3061 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.424335 3126 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:21:58.426542 3104 consensus_queue.cc:1159] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "948356bf66a941ef8249ecbd0ec94f6d" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260924 00:21:58.426393 3126 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.442464 2137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "9e79063532384ada8ea8f8205478adcf"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:21:58.443462 2137 raft_consensus.cc:595] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Received request to transfer leadership to TS 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:58.492694 1892 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:58.508821 1893 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:58.512430 1893 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:58.514750 1893 catalog_manager.cc:5495] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:58.516747 1893 catalog_manager.cc:5495] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 2 to 3, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:58.521121 1893 catalog_manager.cc:5495] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:21:58.536113 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66641de0607c476a958313ea66dff4db"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "9e79063532384ada8ea8f8205478adcf"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:21:58.536824 1990 raft_consensus.cc:595] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Received request to transfer leadership to TS 9e79063532384ada8ea8f8205478adcf
I20260924 00:21:58.568308 2069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:21:58.568915 2069 raft_consensus.cc:595] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:58.569269 2069 raft_consensus.cc:603] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:21:58.571805 2137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:21:58.572808 2137 raft_consensus.cc:595] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.574340 2137 raft_consensus.cc:603] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:21:58.576897 2137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:21:58.577566 2137 raft_consensus.cc:595] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:58.579031 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:21:58.580902 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:21:58.600061 3064 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:58.600721 3064 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.602988 3064 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:58.604643 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:58.604951 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:58.605455 1990 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.607012 2614 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:58.608492 3064 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:58.608911 3064 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:58.609261 3064 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:58.617961 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:21:58.619022 3064 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.621842 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:58.622428 1990 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:58.628418 1989 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.635646 1989 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:58.639904 3126 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:58.638427 1990 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:21:58.648535 2614 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:58.649587 3045 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:58.650100 3045 raft_consensus.cc:686] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:58.651032 3045 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.674207 3103 raft_consensus.cc:979] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 9e79063532384ada8ea8f8205478adcf to start an election
I20260924 00:21:58.674751 3104 raft_consensus.cc:1067] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Signalling peer 9e79063532384ada8ea8f8205478adcf to start an election
I20260924 00:21:58.676729 2642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165"
dest_uuid: "9e79063532384ada8ea8f8205478adcf"
from {username='slave'} at 127.0.0.1:45582
I20260924 00:21:58.677443 2642 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:58.677829 2642 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.679780 3068 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:58.680235 3068 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.684370 3068 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:58.685868 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:58.689639 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:58.690397 1990 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:21:58.691448 2614 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:58.692741 3119 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:21:58.693107 3119 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:58.693466 3119 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:58.696374 3085 raft_consensus.cc:979] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 9e79063532384ada8ea8f8205478adcf to start an election
I20260924 00:21:58.696898 3085 raft_consensus.cc:1067] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Signalling peer 9e79063532384ada8ea8f8205478adcf to start an election
I20260924 00:21:58.698619 2643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66641de0607c476a958313ea66dff4db"
dest_uuid: "9e79063532384ada8ea8f8205478adcf"
from {username='slave'} at 127.0.0.1:45568
I20260924 00:21:58.699163 2643 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:58.699529 2643 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.700743 3119 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.705453 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:58.708660 3119 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:58.709596 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:58.710219 1989 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:21:58.711199 1994 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:58.714895 2642 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.720520 1994 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:21:58.722476 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:58.723274 2069 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.728322 2067 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:58.730468 3104 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:58.734711 2062 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:58.736052 3103 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:58.736989 3064 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:58.739233 2643 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.741555 2643 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:58.741328 1893 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 3 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:58.743409 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:58.746920 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:58.747913 2140 raft_consensus.cc:3032] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.747991 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:58.748438 2140 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:58.747318 2065 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.759634 2140 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.765262 2065 raft_consensus.cc:2454] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.770313 2069 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.775192 2613 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:58.775483 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:58.776239 1994 raft_consensus.cc:3032] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:58.776567 1994 raft_consensus.cc:729] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:58.776517 2140 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.778810 2613 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:58.777449 1994 consensus_queue.cc:260] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.753616 2642 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:58.771437 3061 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:58.753422 3103 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260924 00:21:58.796411 3045 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:58.802210 3045 raft_consensus.cc:686] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:58.803548 1994 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.805357 1992 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.810379 2069 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:58.806483 3068 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:58.811594 3068 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.813640 3068 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:58.781240 3064 raft_consensus.cc:2781] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:58.791536 1990 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.830538 1994 raft_consensus.cc:2454] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.835286 1989 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:21:58.835441 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:58.836220 1995 raft_consensus.cc:2379] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
W20260924 00:21:58.837250 3151 log.cc:924] Time spent T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.091s user 0.002s sys 0.000s
I20260924 00:21:58.822813 3064 raft_consensus.cc:686] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:58.841132 1992 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:58.800485 3061 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.845409 3118 raft_consensus.cc:979] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:21:58.845925 3118 raft_consensus.cc:1067] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:21:58.847111 3061 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:58.847306 2071 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:21:58.842559 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:58.848769 3127 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:21:58.849074 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:58.849581 2069 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.849836 2137 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:58.843276 3064 consensus_queue.cc:237] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.851014 2613 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:58.851150 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:58.851280 1990 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:58.851904 3068 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:58.852161 3061 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:58.852576 3061 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:58.853006 3061 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:58.848986 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:58.817456 3126 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260924 00:21:58.803591 3045 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.862910 1992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:21:58.868844 1992 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:21:58.869824 1992 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.852559 3068 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:58.853976 2614 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:58.855818 2642 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:58.886302 1992 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.873523 3104 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260924 00:21:58.861161 3103 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260924 00:21:58.878089 3035 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:58.896147 3035 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:58.890203 3066 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:21:58.897687 3066 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:58.898860 3066 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:58.883097 3068 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:58.923509 3035 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:58.923837 2065 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:58.925385 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:58.927026 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:58.927635 2065 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:58.929180 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:21:58.939263 3035 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:58.939713 3035 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:58.940100 3035 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:58.896899 2140 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:58.954293 1992 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:58.947300 3103 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260924 00:21:58.992867 3078 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.016244 3078 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.037500 3061 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.037663 3035 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.040468 3035 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.036397 3068 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.044780 3068 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.045825 3061 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:59.050595 3078 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.052372 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:59.053107 2137 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 3
W20260924 00:21:59.058745 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 113307us
I20260924 00:21:59.061503 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:59.062170 2142 raft_consensus.cc:3032] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:21:59.062474 2142 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:59.062906 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:59.064352 3085 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.064086 2065 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:59.064795 3085 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.066655 3085 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.069154 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.069875 2145 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:59.071694 2035 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.071604 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:59.073362 2070 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:21:59.074209 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 60199.7us
I20260924 00:21:59.079890 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:21:59.079051 2137 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:21:59.080552 2643 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.063261 2142 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.082554 2142 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:59.083128 3104 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.083585 3104 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.085489 3104 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.086627 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.087393 1989 raft_consensus.cc:2379] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.088969 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:59.091074 2067 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.093122 2065 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:59.095667 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.072574 3078 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:59.099866 2142 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:59.101272 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:59.101789 2140 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.099376 2070 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:21:59.118067 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.108083 3064 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260924 00:21:59.111730 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.130708 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.131472 2069 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:59.131920 3063 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:59.132663 3063 raft_consensus.cc:686] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:59.133810 3063 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.111936 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.142776 2140 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:21:59.146396 2136 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:59.145391 3064 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:21:59.117671 3085 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.149683 3085 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.151641 3085 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.104822 3078 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.152258 3078 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:59.160701 2643 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:21:59.136162 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.168103 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:59.168617 1991 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:21:59.168889 1994 raft_consensus.cc:2379] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.172184 3035 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:59.172716 3035 raft_consensus.cc:686] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:59.173656 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.173547 3035 consensus_queue.cc:237] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.174311 1989 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:59.175621 1987 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.176865 1995 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:59.145031 3170 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:59.179585 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:59.185377 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.186139 2070 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:59.188406 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.190239 2145 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:21:59.190385 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:21:59.162145 2064 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:59.194051 3068 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:21:59.194101 3064 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:21:59.196512 2613 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:59.198334 1995 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:21:59.208858 3170 raft_consensus.cc:686] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:21:59.211160 3104 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:21:59.211795 3170 consensus_queue.cc:237] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.204726 3064 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:59.214807 3064 raft_consensus.cc:686] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:21:59.215636 3064 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.208735 3126 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.218060 3126 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:59.218396 3126 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.220646 1892 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:59.207458 3040 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.222458 3040 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:59.222808 3040 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.290318 1892 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 4, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 4 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:59.281920 3078 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.264618 3126 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.301400 1892 catalog_manager.cc:5495] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:59.321542 3172 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.322332 3172 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.331727 3172 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.334808 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.335773 2136 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:59.337517 2035 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.338812 3172 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.339345 3172 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:59.339898 3172 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.341528 3103 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:59.342047 3103 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.344861 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.345678 1988 raft_consensus.cc:2354] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4.
I20260924 00:21:59.346843 2110 leader_election.cc:398] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4.
I20260924 00:21:59.347137 2110 leader_election.cc:401] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:21:59.347719 3104 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 4
I20260924 00:21:59.350334 3127 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:21:59.305872 1892 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:21:59.301038 3118 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.303920 3064 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.290562 3040 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.369382 3103 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:59.380168 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:59.380863 1890 catalog_manager.cc:5495] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:21:59.397749 3078 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.399483 3126 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.400231 3126 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.353086 3127 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.400754 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:21:59.424096 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:59.362483 3118 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.427047 3118 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.435704 2070 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.435498 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.437155 2136 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:59.438956 2035 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:21:59.392899 3175 log.cc:924] Time spent T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260924 00:21:59.368288 3063 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.404666 3040 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.464300 3078 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:21:59.363960 3064 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.400853 3126 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:21:59.509569 3040 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:59.477463 3063 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:21:59.525396 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 144769us
I20260924 00:21:59.527885 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:59.528469 1994 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.536563 1994 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:21:59.513384 3119 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260924 00:21:59.521993 3117 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.546231 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.547086 1995 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:21:59.551244 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 348589us
I20260924 00:21:59.552232 1960 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.540576 3035 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:59.516299 3040 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.544250 3184 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:59.558231 3187 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.573277 3103 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.575829 3063 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.577237 3040 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.577790 3169 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.559054 3163 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.570263 3104 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4.
I20260924 00:21:59.569815 3127 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.558739 3117 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:59.596753 2140 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260924 00:21:59.581403 3061 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.584371 3063 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.585753 3163 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:59.568542 3035 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.594909 1989 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260924 00:21:59.581828 3187 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.583956 3103 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:21:59.582506 3184 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.614796 3163 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.611280 3064 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.616864 3118 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.661604 3118 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.663779 3118 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.664283 3118 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.589625 3169 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.666708 1994 raft_consensus.cc:1261] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.669776 3184 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.622135 3126 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.607227 3061 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.682888 3061 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.683713 3061 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.684231 3061 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.613873 3063 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.686941 3035 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.687304 3063 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.630853 3104 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.688170 3104 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.689395 3187 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.690376 3169 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:59.690479 3117 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:21:59.690625 3104 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.648613 3190 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.691485 3190 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.693535 3190 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.695994 3061 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.664695 3118 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.705255 3078 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:21:59.677906 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:59.718575 3203 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.719144 3203 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.721119 3203 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.725935 3203 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.726619 3203 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.727551 3197 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:59.728179 3197 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.739851 1987 raft_consensus.cc:2379] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.741972 2140 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.702889 3064 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260924 00:21:59.740623 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.747380 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.749508 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.752604 1994 raft_consensus.cc:2379] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.731139 3203 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.750355 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.751196 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.756794 1993 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.757426 1991 raft_consensus.cc:2379] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.757965 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.743279 3117 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.741287 3195 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.706629 3159 consensus_queue.cc:1035] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260924 00:21:59.767629 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.768443 2070 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:59.772063 1992 raft_consensus.cc:2379] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.743376 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.715548 3078 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.778925 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.779745 1992 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:59.781771 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.779932 3118 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.774333 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.754024 3188 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.782668 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.783142 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.783069 3188 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.784031 2071 raft_consensus.cc:2354] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3.
I20260924 00:21:59.782702 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.785193 2065 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:59.786083 3190 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.786594 3190 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.788635 3190 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.789283 3197 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.789820 3172 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.791744 3172 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.774626 3117 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.794975 3117 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.759438 3063 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.795974 3063 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.783346 1992 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:59.798418 3063 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.775028 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.801393 1987 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:59.765679 3153 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260924 00:21:59.802109 3195 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.803180 2065 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.804435 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.804527 2062 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:21:59.808703 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.809491 2142 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:59.811295 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.754910 2136 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:21:59.746521 3119 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260924 00:21:59.812767 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.811965 2142 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:21:59.817203 1992 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:59.804208 3195 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.781929 1988 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:59.793081 3172 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.831040 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:21:59.829537 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:21:59.831635 2139 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.816334 3197 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:21:59.821570 3085 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.822764 3203 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.826182 3103 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.844206 2065 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.842854 3197 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.839133 3189 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.853714 3189 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:59.854069 3189 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.829054 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.854650 2035 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:21:59.843703 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.868216 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:21:59.868858 1994 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.866811 2109 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.869304 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.871420 1993 raft_consensus.cc:2379] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.856060 3197 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.850417 3085 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.882757 3085 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.869150 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.883813 1996 raft_consensus.cc:2379] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:21:59.884495 3188 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.884968 1988 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:21:59.886312 2035 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.838214 3172 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.855792 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.877499 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.863715 1987 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:21:59.878526 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:21:59.863951 3103 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.874082 3194 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.869593 3063 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.888441 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.890363 2139 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:21:59.891140 2109 leader_election.cc:398] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3.
I20260924 00:21:59.919955 2109 leader_election.cc:401] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:21:59.921595 2144 raft_consensus.cc:2379] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:59.922556 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.922940 1994 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:21:59.923231 2144 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:21:59.925437 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.926250 2139 raft_consensus.cc:2354] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:21:59.871937 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:21:59.849932 3203 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.895229 3201 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:21:59.874234 2137 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:59.872617 3196 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.915953 3163 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.911170 3066 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260924 00:21:59.929507 3201 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.934155 3203 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.934428 3201 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:21:59.935300 3153 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:21:59.937242 3172 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.937414 3196 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.897681 1991 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:21:59.940001 3196 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.897303 3061 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.915992 2642 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:21:59.926368 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.886206 3159 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260924 00:21:59.917045 3103 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.949455 3103 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:21:59.924906 3063 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:21:59.911362 3194 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:59.958629 3194 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.963414 3208 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:21:59.964035 3208 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:21:59.965994 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:21:59.966499 3208 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.952068 2036 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.970852 2036 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:21:59.992049 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.004654 1995 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:00.005060 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.993948 2143 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:00.005760 2062 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:00.008823 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:21:59.972142 3195 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:00.011525 3195 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:00.012334 3195 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:21:59.972240 3184 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:00.016857 3184 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.018805 3184 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.019483 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.996275 3172 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260924 00:22:00.020089 2145 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:21:59.989105 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:00.022644 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.023259 2070 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:00.012734 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.943287 3061 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:21:59.945762 3187 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:21:59.953054 3063 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:21:59.953588 3190 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.953926 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:21:59.956852 3163 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:21:59.930352 3188 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:21:59.943020 3117 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:21:59.918429 2136 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.020994 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.024303 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.032196 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:00.033481 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.034437 2035 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:00.032811 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:00.035771 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.036257 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.037443 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.035859 3202 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260924 00:21:59.994443 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.029734 3196 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 00:21:59.942003 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 70122.4us
I20260924 00:22:00.067888 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.068768 2067 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:00.072896 2110 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:00.069751 1994 raft_consensus.cc:2354] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:00.069857 2065 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.069948 2145 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.070642 2139 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.075645 2143 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:00.034782 3061 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.071449 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.076896 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.077920 2143 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.079147 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.080595 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.081482 2143 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.082166 2065 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:00.083153 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.072237 2109 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:00.112733 2110 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:00.114343 2110 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.083690 2143 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.111856 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.042472 3117 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:21:59.991519 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:00.123085 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.128423 1994 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:00.119940 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.129292 2142 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.037611 3190 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.131340 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.131978 1994 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:00.133337 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.117499 2139 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:00.136858 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.138501 1960 leader_election.cc:398] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:00.138895 1960 leader_election.cc:401] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:00.082865 3196 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.117918 2137 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:21:59.997603 3189 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:00.144994 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.082748 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.135937 3184 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:00.147183 2144 raft_consensus.cc:2379] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:00.141081 3197 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.151212 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.142649 3126 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.156461 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.137550 3117 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.150257 3205 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:00.036398 2138 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.037324 3187 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.138263 3190 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.136122 3210 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.152230 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.182651 1995 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:00.146175 3196 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.151962 3103 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:00.162372 3197 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.168092 2071 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:00.164736 1891 catalog_manager.cc:5495] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:00.172165 3194 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:00.180449 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.147672 3206 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:00.184397 1996 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:00.214119 2071 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:00.002372 3201 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.167932 3126 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.183102 3198 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.173740 3205 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:00.195555 1988 raft_consensus.cc:2379] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:00.182587 3104 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:00.215147 2642 raft_consensus.cc:2354] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:00.165912 3184 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.141265 3061 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:00.182408 3210 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.194969 3197 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.194307 3168 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:00.226892 3126 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.247526 3198 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.326871 2062 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:00.250519 2069 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:00.316781 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.253614 2063 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:00.245639 3205 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:00.194679 3199 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.261953 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:00.330789 2065 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.329125 2071 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:00.194590 3103 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.340026 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.340628 2071 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:00.257586 3210 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.344233 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.345074 2071 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:00.327392 3187 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.348295 2145 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.328166 3198 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.258011 3104 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.354952 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:00.355618 2143 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.362372 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.386754 2110 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.396160 2065 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:00.375406 3189 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.338263 3168 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.406536 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:00.413702 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.428030 1989 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:00.436545 3198 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.439064 3198 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:00.446987 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 374790us
W20260924 00:22:00.447624 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 373774us
I20260924 00:22:00.450186 2109 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:00.450533 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 416137us
I20260924 00:22:00.422863 3194 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:00.472911 2036 leader_election.cc:398] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:00.473511 2036 leader_election.cc:401] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260924 00:22:00.477727 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 91080.9us
I20260924 00:22:00.426719 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:00.445408 2138 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:00.470992 3201 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.473245 3061 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:22:00.473735 3220 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:00.474071 3184 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:00.555583 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 46617.6us
I20260924 00:22:00.503821 2136 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:00.602385 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 122685us
I20260924 00:22:00.555994 3208 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:00.519711 3103 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:00.610858 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.641870 2071 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:00.642866 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:00.636312 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.639079 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:00.644812 1994 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.668375 2071 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.610381 3194 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.686930 3207 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.641409 3208 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.665776 3206 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:00.705211 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.668298 3104 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:00.515970 3220 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:00.710129 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 258909us
I20260924 00:22:00.711376 2035 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:00.712600 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:00.689605 3194 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.670614 3218 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:00.694597 1994 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:00.673553 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.703274 3208 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:00.727941 2110 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.733232 2109 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:00.733850 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:00.742470 1994 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:00.729429 3194 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.720927 3061 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:00.731797 3195 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:00.818709 3227 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:00.818717 3213 log.cc:924] Time spent T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.349s user 0.000s sys 0.001s
W20260924 00:22:00.786144 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 58302.3us
W20260924 00:22:00.786419 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 53276.6us
I20260924 00:22:00.720213 3229 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:00.769698 2071 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:00.709220 3168 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.818893 3228 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:00.793864 2143 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:00.772511 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:00.838009 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:00.838822 2071 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.820604 3061 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.841882 3226 raft_consensus.cc:2741] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260924 00:22:00.841899 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:00.843226 3221 log.cc:924] Time spent T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20260924 00:22:00.843619 3061 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.820678 3195 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:00.844332 3195 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:00.755132 3103 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:00.846120 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:00.851292 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 138752us
I20260924 00:22:00.858526 1994 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:00.859454 2035 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:00.859669 2110 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.860219 3061 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:00.860590 3061 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:00.860960 3061 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:00.862229 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.862846 1996 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:00.863094 2136 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:00.721069 3206 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.865423 3206 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.819486 3227 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.867388 3061 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.867659 3227 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:00.735387 3218 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.868368 3227 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.868769 3227 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.868980 1996 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:00.869069 3227 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:00.869966 3218 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.871089 3061 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.872602 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:00.872990 2035 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:00.873735 3189 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:00.874097 2035 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:00.874426 3189 raft_consensus.cc:686] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:00.875381 3201 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:00.875784 3201 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:00.876395 3201 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:00.876530 3227 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.875624 3189 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:00.879069 3227 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:00.879760 3061 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:00.863086 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:00.880169 3061 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:00.880594 3061 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:00.880457 1987 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:00.883551 3201 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.885819 3201 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:00.889050 2143 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:00.890858 2035 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:00.899036 2071 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
W20260924 00:22:00.899192 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 170631us
I20260924 00:22:00.899794 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.899968 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:00.903155 3061 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:00.905195 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33238 (request call id 226) took 1531 ms. Trace:
I20260924 00:22:00.905583 1960 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.846009 3229 raft_consensus.cc:686] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:00.924666 2035 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:00.904567 3224 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:00.893395 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:00.898061 3228 raft_consensus.cc:686] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:00.928893 2035 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:00.931897 3217 log.cc:924] Time spent T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.371s user 0.000s sys 0.001s
I20260924 00:22:00.929744 3224 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:00.935443 3219 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20260924 00:22:00.912204 3085 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:00.888809 3103 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:00.931927 3212 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.426s user 0.000s sys 0.001s
I20260924 00:22:00.934582 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:00.910746 3168 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 00:22:00.934893 3223 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
I20260924 00:22:00.935614 3225 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.943195 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:00.943692 3224 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:22:00.938757 3216 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20260924 00:22:00.950471 3201 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:00.960408 3085 raft_consensus.cc:686] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:00.948262 3220 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:00.975898 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 116299us
I20260924 00:22:00.955817 3168 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:00.976963 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:01.013700 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:00.993005 2110 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:00.971796 3206 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:01.020749 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:01.022140 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 72) took 1026 ms. Trace:
I20260924 00:22:01.024473 3237 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:01.025097 3237 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:01.025590 3237 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:22:01.026752 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 97983.6us
I20260924 00:22:01.019721 3040 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:00.905848 1890 rpcz_store.cc:276] 0924 00:21:59.358067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.362675 (+ 4608us) service_pool.cc:227] Handling call
0924 00:22:00.889508 (+1526833us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:21:59.510652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:21:59.560855 (+ 50203us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:21:59.563347 (+ 2492us) write_op.cc:435] Acquired schema lock
0924 00:21:59.563864 (+ 517us) tablet.cc:660] Decoding operations
0924 00:21:59.574256 (+ 10392us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:21:59.574304 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0924 00:21:59.574337 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:21:59.574527 (+ 190us) tablet.cc:699] Row locks acquired
0924 00:21:59.574549 (+ 22us) write_op.cc:260] PREPARE: finished
0924 00:21:59.574688 (+ 139us) write_op.cc:270] Start()
0924 00:21:59.574830 (+ 142us) write_op.cc:276] Timestamp: P: 1790209319574649 usec, L: 0
0924 00:21:59.574854 (+ 24us) op_driver.cc:331] REPLICATION: starting
0924 00:21:59.691122 (+116268us) log.cc:842] Serialized 530 byte log entry
0924 00:21:59.885194 (+194072us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b180016c4e0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d84cc8a8 00007fb6d84cea78 00007fb6d84cea28 00007fb6d84ce9a8 00007fb6d84ce963 00007fb6d84cdc4c 00007fb6daac5206 00007fb6daabd808 00007fb6d4ac8170
0924 00:21:59.885271 (+ 77us) op_driver.cc:444] REPLICATION: finished
0924 00:21:59.895825 (+ 10554us) write_op.cc:301] APPLY: starting
0924 00:21:59.912140 (+ 16315us) tablet.cc:1365] starting BulkCheckPresence
0924 00:21:59.914233 (+ 2093us) tablet.cc:1368] finished BulkCheckPresence
0924 00:21:59.914257 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:21:59.914508 (+ 251us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:21:59.914693 (+ 185us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:21:59.914779 (+ 86us) write_op.cc:312] APPLY: finished
0924 00:21:59.915009 (+ 230us) log.cc:842] Serialized 23 byte log entry
0924 00:21:59.915178 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:21:59.915264 (+ 86us) write_op.cc:454] Released schema lock
0924 00:21:59.915303 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0924 00:21:59.915553 (+ 250us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:21:59.915580 (+ 27us) write_op.cc:454] Released schema lock
0924 00:21:59.992645 (+ 77065us) spinlock_profiling.cc:243] Waited 73 ms on lock 0x7b5400002dd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ab8198 00007fb6d84cbade 00007fb6d84cb2b8 00007fb6d84cfba1 00007fb6d84cfb59 00007fb6d84cfae9 00007fb6d84cfab1 00007fb6d84cedad 00007fb6dadf8204 00007fb6dadf8039
Metrics: {"mutex_wait_us":997,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":321,"apply.run_cpu_time_us":1846,"apply.run_wall_time_us":23588,"num_ops":1,"prepare.queue_time_us":90446,"prepare.run_cpu_time_us":14519,"prepare.run_wall_time_us":498381,"replication_time_us":119461,"spinlock_wait_cycles":177735296}]]}
I20260924 00:22:01.031502 3085 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:01.032120 3211 log.cc:924] Time spent T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.210s user 0.000s sys 0.001s
I20260924 00:22:01.033469 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 74) took 1229 ms. Trace:
I20260924 00:22:00.973541 3225 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:01.032006 3117 consensus_meta.cc:287] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.849s user 0.009s sys 0.000s
I20260924 00:22:01.036087 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:00.975430 3201 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:01.039780 3126 consensus_meta.cc:287] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.703s user 0.007s sys 0.001s
I20260924 00:22:01.023726 3198 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 00:22:01.032034 3222 log.cc:924] Time spent T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.315s user 0.001s sys 0.000s
I20260924 00:22:01.022581 1991 rpcz_store.cc:276] 0924 00:21:59.996075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.006688 (+ 10613us) service_pool.cc:227] Handling call
0924 00:22:00.007139 (+ 451us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:00.029532 (+ 22393us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:00.029565 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:00.029958 (+ 393us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.347683 (+317725us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.347724 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.347792 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.010956 (+663164us) spinlock_profiling.cc:243] Waited 988 us on lock 0x7b48002601d8. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d99637f1 00007fb6d995a8e7 00007fb6d995a77a 00007fb6d9a4e95f 00007fb6d9a4e627 00007fb6d9a62b2c 00007fb6d9a62ad8 00007fb6d9a62a58 00007fb6d9a62a13 00007fb6d9a61cbc 00007fb6d89e3be6
0924 00:22:01.017804 (+ 6848us) raft_consensus.cc:1671] finished
0924 00:22:01.017825 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.022086 (+ 4261us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2216,"raft.run_cpu_time_us":9799,"raft.run_wall_time_us":982316,"spinlock_wait_cycles":2272512,"thread_start_us":279266,"threads_started":1,"wal-append.queue_time_us":440334}
I20260924 00:22:01.051002 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 79) took 1007 ms. Trace:
I20260924 00:22:01.054028 2140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 39) took 1243 ms. Trace:
I20260924 00:22:01.029172 3168 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.020944 3206 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:01.040179 3035 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:01.062530 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:01.054399 2140 rpcz_store.cc:276] 0924 00:21:59.805703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.805855 (+ 152us) service_pool.cc:227] Handling call
0924 00:21:59.806154 (+ 299us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:21:59.813867 (+ 7713us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:21:59.813894 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:21:59.814293 (+ 399us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.338708 (+524415us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.338767 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.338869 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.048590 (+709721us) raft_consensus.cc:1671] finished
0924 00:22:01.048608 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.048718 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7360,"raft.queue_time_us":8164,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":9340,"thread_start_us":498345,"threads_started":1,"wal-append.queue_time_us":599088}
I20260924 00:22:01.049917 3201 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.047960 3040 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:01.114158 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.052673 3198 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:01.023262 3187 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260924 00:22:01.042002 3225 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.054235 2068 rpcz_store.cc:276] 0924 00:22:00.040217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.040343 (+ 126us) service_pool.cc:227] Handling call
0924 00:22:00.040664 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:00.040946 (+ 282us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:00.040979 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:00.042789 (+ 1810us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.440574 (+397785us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.440627 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.440703 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.045789 (+605086us) raft_consensus.cc:1671] finished
0924 00:22:01.045807 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.047363 (+ 1556us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":71350,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":18077,"thread_start_us":313239,"threads_started":1,"wal-append.queue_time_us":432087}
I20260924 00:22:01.115664 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:01.047855 3170 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260924 00:22:01.133663 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 73) took 1101 ms. Trace:
I20260924 00:22:00.965260 3229 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.138181 3236 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:01.112566 3206 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:01.113302 3035 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:01.033847 2064 rpcz_store.cc:276] 0924 00:21:59.804433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.809200 (+ 4767us) service_pool.cc:227] Handling call
0924 00:21:59.845936 (+ 36736us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:21:59.880925 (+ 34989us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:21:59.880965 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:21:59.885329 (+ 4364us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.352349 (+467020us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.353274 (+ 925us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.376884 (+ 23610us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.033202 (+656318us) raft_consensus.cc:1671] finished
0924 00:22:01.033230 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.033393 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6500,"raft.queue_time_us":17078,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"thread_start_us":466640,"threads_started":1,"wal-append.queue_time_us":724221}
I20260924 00:22:01.153762 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:01.144408 1992 rpcz_store.cc:276] 0924 00:22:00.019720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.023921 (+ 4201us) service_pool.cc:227] Handling call
0924 00:22:00.024210 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:00.024503 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:00.024536 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:00.031945 (+ 7409us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.405567 (+373622us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.405620 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.405710 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.120954 (+715244us) raft_consensus.cc:1671] finished
0924 00:22:01.120974 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.121093 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15895,"raft.queue_time_us":214,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":6181,"thread_start_us":336197,"threads_started":1,"wal-append.queue_time_us":644006}
I20260924 00:22:01.126791 3040 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:01.052944 3126 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.045394 3117 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.137481 3198 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.138242 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.215312 3035 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:22:01.161288 3190 consensus_meta.cc:287] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.969s user 0.012s sys 0.000s
I20260924 00:22:01.143162 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:01.161904 3231 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.244s user 0.001s sys 0.001s
I20260924 00:22:01.023430 3228 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.220518 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:01.234302 2068 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:01.230690 2066 raft_consensus.cc:2354] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
W20260924 00:22:01.160178 3188 consensus_meta.cc:287] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 1.110s user 0.007s sys 0.005s
I20260924 00:22:01.206619 3061 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.211928 3040 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.232354 2646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45594 (request call id 241) took 1206 ms. Trace:
I20260924 00:22:01.241631 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:01.242596 2068 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:01.228660 3103 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.248059 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:01.241086 3210 consensus_meta.cc:287] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.888s user 0.008s sys 0.003s
I20260924 00:22:01.228152 2136 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:01.253216 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:01.242293 2646 rpcz_store.cc:276] 0924 00:22:00.025996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.030168 (+ 4172us) service_pool.cc:227] Handling call
0924 00:22:00.068109 (+ 37941us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:00.143995 (+ 75886us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:00.147859 (+ 3864us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:00.165320 (+ 17461us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.403024 (+237704us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.407557 (+ 4533us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.414211 (+ 6654us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.228507 (+814296us) raft_consensus.cc:1671] finished
0924 00:22:01.229683 (+ 1176us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.232264 (+ 2581us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":223,"raft.queue_time_us":8319,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":9906,"thread_start_us":220683,"threads_started":1,"wal-append.queue_time_us":313819}
I20260924 00:22:01.253872 2064 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.240993 3188 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3.
W20260924 00:22:01.247372 3235 log.cc:924] Time spent T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260924 00:22:01.261855 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.221221 3208 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:01.267441 2142 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:01.230181 3190 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.252033 3170 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:01.285867 3199 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:01.288872 2141 raft_consensus.cc:2354] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.286407 3199 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.294265 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 80) took 1221 ms. Trace:
I20260924 00:22:01.300173 2138 raft_consensus.cc:2354] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
W20260924 00:22:01.300704 2642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45594 (request call id 240) took 1335 ms (client timeout 1639 ms). Trace:
I20260924 00:22:01.279093 3230 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:01.286345 2064 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:01.296852 2067 rpcz_store.cc:276] 0924 00:22:00.072793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.073033 (+ 240us) service_pool.cc:227] Handling call
0924 00:22:00.073401 (+ 368us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:00.351520 (+278119us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:00.351549 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:00.409272 (+ 57723us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.691449 (+282177us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.691489 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.691559 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.294047 (+602488us) raft_consensus.cc:1671] finished
0924 00:22:01.294076 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.294209 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":72358,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":7366,"thread_start_us":531006,"threads_started":2,"wal-append.queue_time_us":399195}
I20260924 00:22:01.252825 3210 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:01.310693 3205 consensus_meta.cc:287] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.968s user 0.010s sys 0.001s
W20260924 00:22:01.301321 2642 rpcz_store.cc:269] 0924 00:21:59.965412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.965705 (+ 293us) service_pool.cc:227] Handling call
0924 00:22:01.300481 (+1334776us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540050da54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.300619 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":465312640}
I20260924 00:22:01.319392 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:01.320922 2067 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:22:01.314832 3196 consensus_meta.cc:287] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 1.118s user 0.007s sys 0.004s
W20260924 00:22:01.314356 3197 consensus_meta.cc:287] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.987s user 0.009s sys 0.002s
I20260924 00:22:01.299227 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.303126 3190 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.337956 2140 raft_consensus.cc:2354] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.342949 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33246 (request call id 213) took 1565 ms. Trace:
I20260924 00:22:01.343297 1892 rpcz_store.cc:276] 0924 00:21:59.777326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.777445 (+ 119us) service_pool.cc:227] Handling call
0924 00:21:59.984657 (+207212us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400002dd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41af0 00007fb6d4ac4f2a 00007fb6daa16d6c 00007fb6dab10c3e 00007fb6d8e22fb4 00007fb6d8e22f42 00007fb6d8e22e71 00007fb6d8e22dec 00007fb6d8e220a2 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:01.342892 (+1358235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":134425,"reported_tablets":1,"spinlock_wait_cycles":234055424}
W20260924 00:22:01.344429 3194 consensus_meta.cc:287] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.511s user 0.005s sys 0.004s
I20260924 00:22:01.290629 3170 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:01.337023 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 41) took 1349 ms. Trace:
W20260924 00:22:01.352409 2063 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47766 (request call id 845) took 1314 ms (client timeout 1693 ms). Trace:
W20260924 00:22:01.352814 2063 rpcz_store.cc:269] 0924 00:22:00.038412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.039891 (+ 1479us) service_pool.cc:227] Handling call
0924 00:22:01.352201 (+1312310us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002b2854. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.352321 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":404838784}
I20260924 00:22:01.264758 3238 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.269s
I20260924 00:22:01.349859 2139 rpcz_store.cc:276] 0924 00:21:59.987440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.149865 (+162425us) service_pool.cc:227] Handling call
0924 00:22:00.151332 (+ 1467us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:00.602854 (+451522us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:00.602899 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:00.603439 (+ 540us) log.cc:842] Serialized 26 byte log entry
0924 00:22:00.846985 (+243546us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:00.847032 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:00.847116 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:01.336734 (+489618us) raft_consensus.cc:1671] finished
0924 00:22:01.336763 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:01.336935 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":33405,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":24377,"thread_start_us":600720,"threads_started":2,"wal-append.queue_time_us":286622}
I20260924 00:22:01.354321 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:01.323096 3230 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.344789 3194 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.322758 3205 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.345678 3117 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:01.375980 3242 log.cc:924] Time spent T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20260924 00:22:01.378147 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.378733 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 121) took 1672 ms. Trace:
I20260924 00:22:01.379354 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.349236 3126 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.355054 3188 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:01.379109 1893 rpcz_store.cc:276] 0924 00:21:59.707924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.708105 (+ 181us) service_pool.cc:227] Handling call
0924 00:21:59.997342 (+289237us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400002dd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41af0 00007fb6d4ac4f2a 00007fb6daa16d6c 00007fb6dab10c3e 00007fb6d8e22fb4 00007fb6d8e22f42 00007fb6d8e22e71 00007fb6d8e22dec 00007fb6d8e220a2 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:01.378670 (+1381328us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":259447,"reported_tablets":1,"spinlock_wait_cycles":407388928}
I20260924 00:22:01.385332 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.335857 3196 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.346843 3197 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.371788 1988 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.382037 2144 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:01.406803 3194 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.423985 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47766 (request call id 844) took 1387 ms. Trace:
I20260924 00:22:01.441064 2066 rpcz_store.cc:276] 0924 00:22:00.037052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.038018 (+ 966us) service_pool.cc:227] Handling call
0924 00:22:01.423782 (+1385764us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001ed7d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.423915 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2662416128}
I20260924 00:22:01.444051 2138 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260924 00:22:01.444522 3195 consensus_meta.cc:287] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.600s user 0.004s sys 0.004s
I20260924 00:22:01.445983 1993 raft_consensus.cc:2379] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:01.447408 3197 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.448213 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 968) took 1311 ms. Trace:
I20260924 00:22:01.448621 1993 rpcz_store.cc:276] 0924 00:22:00.136401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.136535 (+ 134us) service_pool.cc:227] Handling call
0924 00:22:01.448002 (+1311467us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540029a9d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.448131 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3008511872}
W20260924 00:22:01.412992 3214 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.500s user 0.004s sys 0.000s
I20260924 00:22:01.454073 2070 raft_consensus.cc:2354] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
W20260924 00:22:01.457074 3247 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260924 00:22:01.461794 3170 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:01.463694 3196 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.464008 3210 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.450559 3208 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.399235 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "150b87da8b7e456cb68c4549975535df"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:01.475593 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47766 (request call id 850) took 1295 ms. Trace:
I20260924 00:22:01.472815 2136 raft_consensus.cc:595] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:01.417478 3253 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:01.456306 2138 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:01.419173 3205 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.475994 2070 rpcz_store.cc:276] 0924 00:22:00.179907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.180076 (+ 169us) service_pool.cc:227] Handling call
0924 00:22:01.475441 (+1295365us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54003f7fd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.475546 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2840435072}
I20260924 00:22:01.459685 3224 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.357304 3237 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.396610 3188 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:01.412343 3201 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.483072 3035 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.485581 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.488333 2137 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:01.490938 2142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 945) took 1118 ms. Trace:
W20260924 00:22:01.491853 2062 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 1039) took 1483 ms (client timeout 1801 ms). Trace:
W20260924 00:22:01.505708 2062 rpcz_store.cc:269] 0924 00:22:00.008525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.008622 (+ 97us) service_pool.cc:227] Handling call
0924 00:22:01.491657 (+1483035us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540014ee54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.491784 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729958144}
I20260924 00:22:01.506755 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60184 (request call id 1319) took 1675 ms. Trace:
I20260924 00:22:01.444936 3195 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.511234 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:01.504746 2142 rpcz_store.cc:276] 0924 00:22:00.374180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.375110 (+ 930us) service_pool.cc:227] Handling call
0924 00:22:00.375465 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:01.270296 (+894831us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:01.490702 (+220406us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b540023c8d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:01.490876 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2050356608}
I20260924 00:22:01.480204 2136 raft_consensus.cc:603] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:01.485740 3230 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.507134 2141 rpcz_store.cc:276] 0924 00:21:59.823480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.823596 (+ 116us) service_pool.cc:227] Handling call
0924 00:22:00.035601 (+212005us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b5800004298. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:01.495208 (+1459607us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54002fe9d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.498531 (+ 3323us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2806563968}
I20260924 00:22:01.515925 2068 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
W20260924 00:22:01.522881 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 485265us
I20260924 00:22:01.509712 3187 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:01.529796 2071 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.534029 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:01.534759 1994 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:01.515473 1990 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:22:01.538784 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 488213us
I20260924 00:22:01.523473 3188 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:01.550474 3035 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.562875 1994 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:01.566146 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:01.543983 2071 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:01.526401 3040 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.509972 3253 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.555778 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:01.585474 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.589601 2142 raft_consensus.cc:2354] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.537077 3187 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.598964 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.600385 2141 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.601925 2110 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:01.607628 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.609616 2142 raft_consensus.cc:2354] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.587168 3199 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.611280 3250 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:01.613320 1996 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:01.595824 3190 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:01.617471 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.603193 1987 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.621079 2035 leader_election.cc:398] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667) with higher term. Message: Invalid argument: T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
W20260924 00:22:01.619314 3207 consensus_meta.cc:287] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.928s user 0.011s sys 0.003s
I20260924 00:22:01.648231 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:01.613200 3210 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
W20260924 00:22:01.658969 2109 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.2:39591 timed out after 1.801s (SENT) [suppressed 4 similar messages]
I20260924 00:22:01.661187 2138 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.611984 3250 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:01.663872 2035 leader_election.cc:398] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:01.677615 2035 leader_election.cc:401] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260924 00:22:01.661387 1992 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.681330 2035 leader_election.cc:398] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.681814 2035 leader_election.cc:401] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260924 00:22:01.659721 3236 raft_consensus.cc:686] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:01.682596 2035 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:01.630194 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 507971us
I20260924 00:22:01.660336 3201 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.654546 3224 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.664937 3207 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.700225 3253 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.700362 2063 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:01.673589 3188 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:01.630313 3190 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.701992 2035 leader_election.cc:398] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.705693 2035 leader_election.cc:401] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:01.678423 3237 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.684902 3218 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.683851 3195 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.686733 3205 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:01.683501 3232 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:01.711061 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 153523us
I20260924 00:22:01.683531 3210 raft_consensus.cc:686] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:01.704440 3198 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.718092 3224 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.720755 2141 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:01.704651 3249 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:01.686600 3203 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260924 00:22:01.704574 3196 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260924 00:22:01.732146 3168 consensus_meta.cc:287] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.618s user 0.008s sys 0.003s
I20260924 00:22:01.719633 2035 leader_election.cc:398] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.739449 2035 leader_election.cc:401] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260924 00:22:01.735814 3040 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.736374 3250 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:01.734130 3206 consensus_meta.cc:287] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.524s user 0.011s sys 0.000s
I20260924 00:22:01.736276 3236 consensus_queue.cc:237] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.743096 2110 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:01.743495 2035 leader_election.cc:398] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.782827 2035 leader_election.cc:401] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260924 00:22:01.747298 1992 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:01.706849 3188 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.741904 3187 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:01.738421 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:01.737987 3232 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.767252 3206 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.774405 3190 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.775660 3233 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 00:22:01.787828 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 44812.8us
I20260924 00:22:01.795977 2110 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:01.789080 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 53447.9us
W20260924 00:22:01.781250 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.028s sys 0.094s
I20260924 00:22:01.747987 3063 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.792243 1992 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260924 00:22:01.798237 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.749s user 0.028s sys 0.094s
W20260924 00:22:01.800717 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 753226us
I20260924 00:22:01.766928 1989 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:01.803658 3232 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.742689 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:01.793080 3078 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.814687 1991 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.748334 3168 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.748139 3249 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.794180 3118 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:01.820611 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:01.825788 1992 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:01.810984 3225 consensus_meta.cc:287] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.646s user 0.003s sys 0.011s
I20260924 00:22:01.799809 3238 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:01.828864 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.810043 3233 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:01.809839 3201 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:01.830467 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:01.830711 2141 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.829233 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:01.831207 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:01.819509 3040 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:01.841518 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:01.839282 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:01.845512 3251 log.cc:924] Time spent T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260924 00:22:01.845764 2066 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:01.818984 3199 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:01.853379 3254 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:01.847851 2145 raft_consensus.cc:2354] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:01.818346 3172 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:01.852991 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:01.823275 3196 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:01.824002 3195 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:01.832000 3118 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:01.899576 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:01.835613 3233 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.900134 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.901517 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:01.859488 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 200530us
I20260924 00:22:01.799530 3210 consensus_queue.cc:237] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:01.856150 3207 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:01.823899 3035 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260924 00:22:01.845033 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:01.916442 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:01.918970 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:01.836184 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:01.851692 3238 raft_consensus.cc:686] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:01.922420 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 1048) took 1196 ms. Trace:
I20260924 00:22:01.899150 3118 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:01.920997 1893 catalog_manager.cc:5495] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 3, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 3 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:01.863457 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:01.838815 3225 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.849116 3201 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.853941 3254 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:01.898917 3172 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.047281 1992 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.046579 2068 rpcz_store.cc:276] 0924 00:22:00.717667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.247809 (+530142us) service_pool.cc:227] Handling call
0924 00:22:01.914148 (+666339us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b5400347e54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.914307 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":614261120}
I20260924 00:22:02.053448 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.053726 2035 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:02.054425 2141 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:02.054051 1992 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:02.062927 2109 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:01.909797 3170 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:02.064093 2109 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:02.066682 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:02.068799 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:02.027720 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 215240us
I20260924 00:22:01.925123 3237 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 00:22:02.026944 3263 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260924 00:22:02.065382 3241 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:02.092229 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 1046) took 1375 ms. Trace:
W20260924 00:22:02.075162 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 53923.5us
I20260924 00:22:02.029085 3078 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.935053 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:02.083935 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1050) took 1365 ms. Trace:
I20260924 00:22:02.106317 2063 rpcz_store.cc:276] 0924 00:22:00.718326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.353968 (+635642us) service_pool.cc:227] Handling call
0924 00:22:01.492252 (+138284us) spinlock_profiling.cc:243] Waited 9.64 ms on lock 0x7b580008b898. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:01.492477 (+ 225us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:01.700975 (+208498us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:02.083726 (+382751us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54001ec3d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:02.083880 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":499264512}
I20260924 00:22:02.077572 1991 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.092756 2071 rpcz_store.cc:276] 0924 00:22:00.716913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.934232 (+217319us) service_pool.cc:227] Handling call
0924 00:22:02.092006 (+1157774us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54000b1b54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:02.092153 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1324677376}
I20260924 00:22:02.110096 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:02.084595 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.111346 1995 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:02.077046 2110 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:02.085824 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.114959 1993 raft_consensus.cc:2379] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:02.086952 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:02.024291 3063 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.959848 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:02.033041 3218 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.031837 3188 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:02.124137 3188 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:02.124593 3188 raft_consensus.cc:686] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:02.125692 1992 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.125819 3228 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:02.125447 3188 consensus_queue.cc:237] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:02.126350 3228 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:02.127144 3228 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.127942 2066 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.128506 2070 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:02.129534 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:01.937186 3198 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:02.130726 1995 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.089684 2143 raft_consensus.cc:2354] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:01.922559 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 946) took 1544 ms. Trace:
I20260924 00:22:02.131901 2137 rpcz_store.cc:276] 0924 00:22:00.375968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.513956 (+137988us) service_pool.cc:227] Handling call
0924 00:22:01.278903 (+764947us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b5800004298. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:01.486108 (+207205us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:01.507549 (+ 21441us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:01.862527 (+354978us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54000e0bd0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abba09 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:01.920704 (+ 58177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":484490496}
I20260924 00:22:01.913580 3195 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.074885 2138 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:02.134608 3195 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.025246 3234 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:02.135305 3234 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.137202 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.137702 3234 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.026090 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:02.138484 2136 raft_consensus.cc:595] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:02.138875 2136 raft_consensus.cc:603] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:01.919852 3196 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.142491 3196 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:02.142411 3241 raft_consensus.cc:686] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:02.143813 3241 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.065475 3199 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:02.146860 3199 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.147347 3238 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.118566 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:02.118820 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:02.117671 3250 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260924 00:22:02.120060 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:02.091312 3249 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.130581 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:02.074360 3230 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:02.119321 3172 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.130663 2141 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:02.156754 2067 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
W20260924 00:22:02.098354 1988 consensus_meta.cc:287] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.643s user 0.003s sys 0.009s
I20260924 00:22:01.937606 3207 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.172863 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:02.173089 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:02.184439 3040 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:02.075096 3170 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:02.365957 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 301907us
I20260924 00:22:02.457767 2109 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:02.075409 1990 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.430878 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:02.341252 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 441733us
I20260924 00:22:02.332863 3206 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.190277 2143 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60184 (request call id 1340) took 1153 ms. Trace:
I20260924 00:22:02.469720 2143 rpcz_store.cc:276] 0924 00:22:01.037355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.379101 (+341746us) service_pool.cc:227] Handling call
0924 00:22:02.190057 (+810956us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b5400450e54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:02.190212 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1631747840}
I20260924 00:22:02.420648 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:02.470954 2065 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:02.434063 2063 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.461576 2071 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:02.466784 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:02.477036 1996 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:02.186012 3199 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:02.220969 3205 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:02.450338 3040 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:02.188515 3201 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.191025 3168 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.187259 3225 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:02.331972 3153 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:02.465389 3224 consensus_meta.cc:287] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.730s user 0.011s sys 0.004s
W20260924 00:22:02.238095 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 63999.5us
I20260924 00:22:02.451253 1988 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.464562 3188 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:02.479887 3169 raft_consensus.cc:2741] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260924 00:22:02.517252 2071 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.504444 2065 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:02.506343 3224 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.506801 3153 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.509060 1960 leader_election.cc:398] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
W20260924 00:22:02.509711 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 49028.2us
I20260924 00:22:02.474655 3172 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:02.481078 3275 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:02.519954 2068 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:02.484166 3258 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:02.522670 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:02.523211 1960 leader_election.cc:401] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:02.479470 3244 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260924 00:22:02.529503 1960 leader_election.cc:398] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:02.529953 1960 leader_election.cc:401] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260924 00:22:02.524626 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 447677us
I20260924 00:22:02.519505 3061 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:02.523171 1988 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 971) took 1882 ms (client timeout 1816 ms). Trace:
I20260924 00:22:02.526760 3172 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.572074 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.535696 3061 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.578248 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.621444 1992 raft_consensus.cc:2379] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
W20260924 00:22:02.580582 3280 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260924 00:22:02.529286 3275 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:02.586401 3276 log.cc:924] Time spent T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
W20260924 00:22:02.595184 1987 consensus_meta.cc:287] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.915s user 0.008s sys 0.003s
I20260924 00:22:02.589702 3273 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260924 00:22:02.602010 1995 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:02.634035 1987 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.636241 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 987) took 1110 ms. Trace:
I20260924 00:22:02.636579 1987 rpcz_store.cc:276] 0924 00:22:01.525665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.529079 (+ 3414us) service_pool.cc:227] Handling call
0924 00:22:02.636182 (+1107103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:02.531260 3188 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:02.531404 3169 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:02.537276 2066 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:02.615335 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:02.615031 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:02.648783 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 110947us
I20260924 00:22:02.535939 3168 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:02.601392 3199 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:02.629980 2110 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:02.606870 3225 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:02.613901 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 78373.2us
I20260924 00:22:02.622486 3274 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:02.531786 3258 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.617940 3170 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.619650 3207 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.643438 3275 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.617717 3228 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
W20260924 00:22:02.615674 1988 rpcz_store.cc:269] 0924 00:22:00.637436 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:00.640284 (+ 2848us) service_pool.cc:227] Handling call
0924 00:22:01.354171 (+713887us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:02.519791 (+1165620us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27457792}
I20260924 00:22:02.652709 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:02.676259 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.630618 3190 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:02.636884 3061 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:02.654726 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:02.649438 3118 consensus_meta.cc:287] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.584s user 0.010s sys 0.000s
I20260924 00:22:02.669865 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:02.676959 1995 raft_consensus.cc:2354] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.653944 3232 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:02.652506 3187 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:02.683364 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.676626 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.685497 1995 raft_consensus.cc:2354] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.654325 3188 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:02.686218 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.677544 3277 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:02.688177 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:02.667086 3172 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:02.690536 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.690466 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.671638 3153 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.710893 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:02.705034 1992 raft_consensus.cc:2354] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
W20260924 00:22:02.702597 3233 consensus_meta.cc:287] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.778s user 0.000s sys 0.011s
I20260924 00:22:02.656287 3225 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.713579 2062 raft_consensus.cc:2354] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:02.713495 2036 leader_election.cc:398] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.706432 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.722059 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.722509 3225 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:02.691545 3277 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:02.706821 3040 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.707777 3188 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.667265 3199 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.672783 3274 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.681543 3170 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:02.715366 2036 leader_election.cc:398] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.685268 3230 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 00:22:02.799779 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 84635.9us
I20260924 00:22:02.789556 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.761920 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:02.681810 3249 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260924 00:22:02.802210 1992 raft_consensus.cc:2354] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.810034 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:02.693202 3187 raft_consensus.cc:686] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:02.811004 2062 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:02.815408 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.819813 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:02.799499 3277 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:02.687208 3118 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.688076 3190 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.723266 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.710373 3061 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.108s
I20260924 00:22:02.831534 3241 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:02.842564 3230 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:02.843187 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.737089 3233 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.851264 2139 raft_consensus.cc:2354] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:02.851061 3187 consensus_queue.cc:237] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.851792 3118 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:02.851627 3241 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.852568 3236 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:02.867524 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 187792us
I20260924 00:22:02.870014 2110 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:02.872444 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.875883 2110 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:02.890106 3274 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:02.877333 2069 raft_consensus.cc:1226] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260924 00:22:02.877331 3118 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:02.891844 3040 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.894568 3169 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:02.895486 3118 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:02.895664 3238 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:02.896354 3238 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:02.896483 3118 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.897447 2064 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:02.897475 3238 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.908385 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33232 (request call id 214) took 3345 ms. Trace:
I20260924 00:22:02.908911 3061 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:02.847903 3287 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:02.904345 2110 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:02.912547 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.878777 3241 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:02.928174 3061 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:02.929224 3225 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:02.929816 3225 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:02.930143 3254 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.930198 3225 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:02.917358 3258 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:02.915712 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 994) took 1321 ms. Trace:
I20260924 00:22:02.931507 1994 rpcz_store.cc:276] 0924 00:22:01.594059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.854383 (+260324us) service_pool.cc:227] Handling call
0924 00:22:01.854646 (+ 263us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:02.445409 (+590763us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:02.452224 (+ 6815us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:02.462393 (+ 10169us) log.cc:842] Serialized 26 byte log entry
0924 00:22:02.525864 (+ 63471us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:02.525909 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:02.915332 (+389423us) spinlock_profiling.cc:243] Waited 44.6 ms on lock 0x7b540004bad4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:02.915390 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:02.915436 (+ 46us) raft_consensus.cc:1671] finished
0924 00:22:02.915450 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:02.915654 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9173,"raft.queue_time_us":19447,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6079,"spinlock_wait_cycles":102648704,"thread_start_us":400391,"threads_started":2,"wal-append.queue_time_us":88646}
I20260924 00:22:02.932941 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57748 (request call id 961) took 1102 ms. Trace:
I20260924 00:22:02.933413 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.933298 2139 rpcz_store.cc:276] 0924 00:22:01.830975 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.831078 (+ 103us) service_pool.cc:227] Handling call
0924 00:22:02.086789 (+255711us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b5800004298. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:02.932768 (+845979us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b540008f054. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:02.932894 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1691936384}
W20260924 00:22:02.934478 1991 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 979) took 1912 ms (client timeout 2383 ms). Trace:
I20260924 00:22:02.934537 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.875016 3236 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:02.934930 1991 rpcz_store.cc:269] 0924 00:22:01.022110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.062194 (+ 40084us) service_pool.cc:227] Handling call
0924 00:22:02.934335 (+1872141us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b54000749d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:02.934424 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1619128704}
I20260924 00:22:02.930442 3061 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:02.935921 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:02.936596 1991 raft_consensus.cc:2354] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:02.938087 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 1060) took 1086 ms. Trace:
I20260924 00:22:02.938416 2067 rpcz_store.cc:276] 0924 00:22:01.852143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.852714 (+ 571us) service_pool.cc:227] Handling call
0924 00:22:02.937940 (+1085226us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b5400120554. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:02.938040 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":654376192}
I20260924 00:22:02.944303 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:02.866008 3168 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:02.945231 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:02.928489 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.946491 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.948314 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:02.942703 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:02.935303 1994 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:02.861020 3230 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:02.944670 3061 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:02.975828 1995 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:02.962337 2110 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:03.442162 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 479900us
I20260924 00:22:03.443912 2110 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:03.435804 1991 raft_consensus.cc:1226] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3. Ops: []
W20260924 00:22:03.080802 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 349196us
I20260924 00:22:03.470006 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:03.373418 3236 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.944692 3287 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.019534 3172 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.473085 3172 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.458624 3230 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:03.474363 3230 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:03.474936 3230 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:03.475328 3230 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:03.019361 3168 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:03.475543 3172 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:02.968593 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:03.013283 3170 raft_consensus.cc:686] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:03.477543 3233 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:03.477837 2614 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:03.448944 3190 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:03.477916 3170 consensus_queue.cc:237] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.479146 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:03.480906 3064 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 00:22:03.482203 1988 consensus_meta.cc:287] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.576s user 0.008s sys 0.000s
I20260924 00:22:03.482688 1988 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:03.463974 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 86) took 1618 ms. Trace:
I20260924 00:22:03.506978 2070 rpcz_store.cc:276] 0924 00:22:01.845696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.527246 (+681550us) service_pool.cc:227] Handling call
0924 00:22:02.527524 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:02.638195 (+110671us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:02.638238 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:02.638292 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:02.638332 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.463668 (+825336us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54003f7ad4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:03.463739 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.463909 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":254018304}
I20260924 00:22:03.372617 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:03.223664 3254 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:03.120572 3274 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.211637 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:03.297729 3279 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:02.950862 3258 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.514341 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 89) took 1411 ms. Trace:
I20260924 00:22:03.297070 3199 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:03.109514 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 460462us
I20260924 00:22:03.460162 3188 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
W20260924 00:22:03.511231 2062 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.007s sys 0.012s
I20260924 00:22:03.121610 3292 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:03.479266 3243 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:03.435525 3241 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:02.981736 3250 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:03.475944 3168 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.021461 3210 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.637s
I20260924 00:22:03.517263 2066 rpcz_store.cc:276] 0924 00:22:02.103153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.695513 (+592360us) service_pool.cc:227] Handling call
0924 00:22:02.695770 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:02.695952 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 5
0924 00:22:03.077071 (+381119us) log.cc:842] Serialized 14 byte log entry
0924 00:22:03.514006 (+436935us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:03.514067 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:03.514110 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.514177 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.514284 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20398,"prepare.queue_time_us":940715,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":20490,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":769720,"threads_started":2}
W20260924 00:22:03.673110 2108 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.538s user 0.013s sys 0.040s
I20260924 00:22:03.514806 1960 leader_election.cc:398] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:03.674463 1960 leader_election.cc:401] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260924 00:22:03.675181 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 160499us
I20260924 00:22:03.675896 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:03.678586 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:03.680308 2062 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s user 0.012s sys 0.017s
W20260924 00:22:03.498250 3256 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.008s sys 0.027s
I20260924 00:22:03.681061 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 90) took 1577 ms. Trace:
I20260924 00:22:03.044278 3169 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.020669 3208 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:03.729487 2062 rpcz_store.cc:276] 0924 00:22:02.103610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.816027 (+712417us) service_pool.cc:227] Handling call
0924 00:22:02.816319 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:02.816530 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 5
0924 00:22:02.969794 (+153264us) log.cc:842] Serialized 14 byte log entry
0924 00:22:03.680792 (+710998us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:03.680851 (+ 59us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:03.680888 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.680935 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.681021 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6030,"prepare.queue_time_us":824224,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":17961,"raft.queue_time_us":79,"thread_start_us":834653,"threads_started":2,"wal-append.queue_time_us":694969}
W20260924 00:22:03.730672 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 52185us
W20260924 00:22:03.664673 3225 consensus_meta.cc:287] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.734s user 0.006s sys 0.004s
W20260924 00:22:03.523861 3277 consensus_meta.cc:287] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.676s user 0.009s sys 0.001s
I20260924 00:22:03.511760 3273 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:03.553754 3233 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:02.925158 1891 rpcz_store.cc:276] 0924 00:21:59.563211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:59.566861 (+ 3650us) service_pool.cc:227] Handling call
0924 00:21:59.672521 (+105660us) spinlock_profiling.cc:243] Waited 79.6 ms on lock 0x7b5400002dd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41af0 00007fb6d4ac4f2a 00007fb6daa16d6c 00007fb6dab10c3e 00007fb6d8e22fb4 00007fb6d8e22f42 00007fb6d8e22e71 00007fb6d8e22dec 00007fb6d8e220a2 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:02.908288 (+3235767us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:01.040352 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:01.281630 (+241278us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:01.287095 (+ 5465us) write_op.cc:435] Acquired schema lock
0924 00:22:01.289362 (+ 2267us) tablet.cc:660] Decoding operations
0924 00:22:01.338264 (+ 48902us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:01.355484 (+ 17220us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:01.358960 (+ 3476us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:01.380228 (+ 21268us) tablet.cc:699] Row locks acquired
0924 00:22:01.380252 (+ 24us) write_op.cc:260] PREPARE: finished
0924 00:22:01.380423 (+ 171us) write_op.cc:270] Start()
0924 00:22:01.380544 (+ 121us) write_op.cc:276] Timestamp: P: 1790209321380365 usec, L: 0
0924 00:22:01.380566 (+ 22us) op_driver.cc:331] REPLICATION: starting
0924 00:22:01.381304 (+ 738us) log.cc:842] Serialized 535 byte log entry
0924 00:22:02.150393 (+769089us) op_driver.cc:444] REPLICATION: finished
0924 00:22:02.509468 (+359075us) write_op.cc:301] APPLY: starting
0924 00:22:02.509547 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:02.511354 (+ 1807us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:02.525610 (+ 14256us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:02.535054 (+ 9444us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:02.571460 (+ 36406us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:02.571509 (+ 49us) write_op.cc:312] APPLY: finished
0924 00:22:02.571714 (+ 205us) log.cc:842] Serialized 23 byte log entry
0924 00:22:02.591183 (+ 19469us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:02.591265 (+ 82us) write_op.cc:454] Released schema lock
0924 00:22:02.591302 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:02.591597 (+ 295us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:02.591628 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":793574,"reported_tablets":3,"spinlock_wait_cycles":183128320,"child_traces":[["op",{"apply.queue_time_us":286432,"apply.run_cpu_time_us":8420,"apply.run_wall_time_us":82395,"num_ops":1,"prepare.queue_time_us":115030,"prepare.run_cpu_time_us":6233,"prepare.run_wall_time_us":533464,"raft.queue_time_us":468185,"raft.run_cpu_time_us":15754,"raft.run_wall_time_us":403867,"replication_time_us":735544,"thread_start_us":611435,"threads_started":4,"wal-append.queue_time_us":170419}]]}
W20260924 00:22:03.758598 2111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.015s sys 0.009s
W20260924 00:22:03.810982 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.704s user 0.015s sys 0.010s
I20260924 00:22:03.767480 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 93) took 1078 ms. Trace:
W20260924 00:22:03.768744 2064 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 1058) took 1921 ms (client timeout 2186 ms). Trace:
I20260924 00:22:03.769826 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 860) took 1946 ms. Trace:
I20260924 00:22:03.778477 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1056) took 1931 ms. Trace:
I20260924 00:22:03.789642 3228 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:03.790822 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1350) took 1105 ms. Trace:
I20260924 00:22:03.795923 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1347) took 1119 ms. Trace:
I20260924 00:22:03.798153 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 92) took 1644 ms. Trace:
W20260924 00:22:03.735519 3256 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.798s user 0.012s sys 0.028s
I20260924 00:22:03.804512 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1345) took 1128 ms. Trace:
W20260924 00:22:03.819933 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 375024us
I20260924 00:22:03.847716 1993 rpcz_store.cc:276] 0924 00:22:02.676144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.676291 (+ 147us) service_pool.cc:227] Handling call
0924 00:22:03.804241 (+1127950us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002e9354. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:03.804436 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2487427840}
I20260924 00:22:03.641029 3275 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.642536 3265 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.628108 3294 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.677989 3296 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:03.859611 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:03.628165 3253 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:03.511528 3215 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:03.799412 3287 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:03.806565 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:03.819547 1995 rpcz_store.cc:276] 0924 00:22:02.685682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.686013 (+ 331us) service_pool.cc:227] Handling call
0924 00:22:03.790593 (+1104580us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b540036f1d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:03.790740 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":611640064}
I20260924 00:22:03.542587 3258 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.733893 3199 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:03.483881 3061 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.733882 3188 raft_consensus.cc:686] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:22:03.883185 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 205846us
I20260924 00:22:03.855391 3249 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:03.873745 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:03.874346 2036 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.023s sys 0.078s
I20260924 00:22:03.734582 3279 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:03.766450 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 285917us
I20260924 00:22:03.542023 3274 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.523415 3254 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.678009 3300 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:03.809736 2643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 262) took 1721 ms. Trace:
I20260924 00:22:03.686708 3241 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.884479 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:03.728391 3250 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.851365 3207 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:03.808883 3277 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:03.887667 2036 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.536s user 0.023s sys 0.079s
I20260924 00:22:03.853736 2063 rpcz_store.cc:276] 0924 00:22:02.153603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.820713 (+667110us) service_pool.cc:227] Handling call
0924 00:22:02.846354 (+ 25641us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:02.877603 (+ 31249us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:02.877634 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:02.877670 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:02.877697 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.797754 (+920057us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b54004290d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:03.797819 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.798073 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71461760}
I20260924 00:22:03.765712 3169 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.687386 3292 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.808514 3208 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.819634 3233 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:03.901007 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 142479us
I20260924 00:22:03.885939 3307 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:03.893926 3299 log.cc:924] Time spent T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.223s user 0.000s sys 0.001s
I20260924 00:22:03.910858 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:03.860971 3275 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:03.914439 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 610955us
I20260924 00:22:03.847581 1988 rpcz_store.cc:276] 0924 00:22:02.676718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.687937 (+ 11219us) service_pool.cc:227] Handling call
0924 00:22:03.795685 (+1107748us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54000050d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:03.795832 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449378176}
I20260924 00:22:03.918601 2109 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:03.817849 2071 rpcz_store.cc:276] 0924 00:22:01.847038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.777421 (+1930383us) service_pool.cc:227] Handling call
0924 00:22:03.777863 (+ 442us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:03.778073 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:03.778107 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:03.778159 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:03.778203 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.778265 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.778406 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260924 00:22:03.817509 2064 rpcz_store.cc:269] 0924 00:22:01.847763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.118567 (+270804us) service_pool.cc:227] Handling call
0924 00:22:03.768520 (+1649953us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b5400468a54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:03.768679 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":788390656}
I20260924 00:22:03.807535 3225 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.878609 3253 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.930115 2065 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:03.872656 3265 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.934182 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:03.934343 3253 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:03.934384 3265 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:03.934737 3234 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:03.935251 3234 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.936738 3225 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:03.938148 3225 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:03.938836 3225 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.938041 3234 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:03.943012 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:03.944629 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:03.944833 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:03.947033 2142 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:03.886349 3199 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.952468 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 88) took 2106 ms. Trace:
I20260924 00:22:03.952864 3199 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:03.902436 1989 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1342) took 1227 ms (client timeout 1552 ms). Trace:
I20260924 00:22:03.952855 2065 rpcz_store.cc:276] 0924 00:22:01.846686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.107472 (+1260786us) service_pool.cc:227] Handling call
0924 00:22:03.107879 (+ 407us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:03.933234 (+825355us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.952250 (+ 19016us) spinlock_profiling.cc:243] Waited 822 ms on lock 0x7b54001568d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:03.952414 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1890120064}
I20260924 00:22:03.872449 3294 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.819128 2069 rpcz_store.cc:276] 0924 00:22:01.823108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.824429 (+ 1321us) service_pool.cc:227] Handling call
0924 00:22:01.854380 (+ 29951us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:02.891305 (+1036925us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.769506 (+878201us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001563d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:03.769761 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2347255552}
I20260924 00:22:03.912065 3210 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:03.894888 3249 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.956725 1960 leader_election.cc:398] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:03.960850 3199 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
W20260924 00:22:03.961592 3304 log.cc:924] Time spent T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.279s user 0.000s sys 0.001s
I20260924 00:22:03.961947 3318 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:03.962261 3306 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260924 00:22:03.962906 3309 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20260924 00:22:03.963542 3305 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260924 00:22:03.913015 3228 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:03.913187 3310 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:03.917810 3300 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:03.947535 3225 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:03.969154 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 1361) took 1276 ms. Trace:
I20260924 00:22:03.972914 2036 leader_election.cc:398] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:03.965891 3308 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:04.027482 1992 rpcz_store.cc:276] 0924 00:22:02.692873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.219800 (+526927us) service_pool.cc:227] Handling call
0924 00:22:03.461750 (+241950us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:03.483758 (+ 22008us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:03.483786 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:03.511672 (+ 27886us) log.cc:842] Serialized 26 byte log entry
0924 00:22:03.860054 (+348382us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:03.860109 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.860202 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:03.968896 (+108694us) raft_consensus.cc:1671] finished
0924 00:22:03.968924 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.969054 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21594,"raft.queue_time_us":22116,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":11666,"thread_start_us":297536,"threads_started":1,"wal-append.queue_time_us":408994}
I20260924 00:22:03.982535 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.033159 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.033301 2070 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:03.982987 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:03.984403 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:03.986366 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:03.929544 3170 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:03.999389 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.039423 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:04.001745 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.005270 1960 leader_election.cc:398] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:04.009770 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.010056 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:04.012217 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:03.921116 3207 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.056912 3277 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.022219 3195 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:04.057713 3195 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.058235 3207 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:04.028781 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 56062.5us
I20260924 00:22:04.059013 3279 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.059736 3195 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.006202 3253 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:04.060462 3253 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.930058 3307 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:03.878439 3215 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.995362 3168 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.020608 1994 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:04.031287 3228 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.034794 2142 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:04.041363 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.076105 1992 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:04.077024 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.077276 1992 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260924 00:22:04.078383 1993 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:04.046247 1960 leader_election.cc:401] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260924 00:22:04.033936 2071 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:03.988485 3311 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:04.034404 3301 log.cc:924] Time spent T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.367s user 0.001s sys 0.001s
I20260924 00:22:04.003819 3199 raft_consensus.cc:686] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:03.965284 3295 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20260924 00:22:03.988096 3244 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260924 00:22:04.061061 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:04.045028 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 54) took 1426 ms. Trace:
I20260924 00:22:03.977952 3210 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:04.044636 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:04.100795 3274 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.044946 3243 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:04.090121 2036 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:03.965891 1989 rpcz_store.cc:269] 0924 00:22:02.675189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.680274 (+ 5085us) service_pool.cc:227] Handling call
0924 00:22:03.806325 (+1126051us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:03.900676 (+ 94351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45404416}
I20260924 00:22:04.045185 3300 raft_consensus.cc:686] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:04.103116 2036 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:04.103348 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 881) took 1385 ms. Trace:
I20260924 00:22:04.093436 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:04.104230 2036 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:04.034387 2145 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:04.101374 3061 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.103827 2067 rpcz_store.cc:276] 0924 00:22:02.718295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.952838 (+234543us) service_pool.cc:227] Handling call
0924 00:22:02.953277 (+ 439us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:02.953698 (+ 421us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:02.953737 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:02.961735 (+ 7998us) log.cc:842] Serialized 26 byte log entry
0924 00:22:03.513779 (+552044us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:03.514398 (+ 619us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.514479 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:04.103093 (+588614us) raft_consensus.cc:1671] finished
0924 00:22:04.103120 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:04.103281 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65221,"raft.queue_time_us":527,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":41001,"thread_start_us":472026,"threads_started":1,"wal-append.queue_time_us":773610}
I20260924 00:22:03.966816 3188 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:04.080937 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 75905.9us
I20260924 00:22:03.902807 2643 rpcz_store.cc:276] 0924 00:22:02.145865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.164705 (+ 18840us) service_pool.cc:227] Handling call
0924 00:22:02.465627 (+300922us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:02.535728 (+ 70101us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:02.571690 (+ 35962us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:02.590601 (+ 18911us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:02.590662 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.682289 (+1091627us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x7b54003183d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:03.729433 (+ 47144us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.765521 (+ 36088us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117369216}
I20260924 00:22:04.108811 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:04.013427 3318 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:04.085373 3168 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:04.110152 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.074740 2144 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:04.084807 3253 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:04.122824 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.128158 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 122) took 2602 ms. Trace:
I20260924 00:22:04.140715 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.035274 3310 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.080286 1988 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:04.137458 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.097009 3234 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:04.139613 1992 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:04.130210 1892 catalog_manager.cc:5495] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:04.142652 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.145006 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.106941 3195 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:04.045394 3308 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.107112 3296 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:04.134124 3215 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.134698 3250 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:04.111533 3258 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.113478 3061 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:04.134773 3274 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:04.153760 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.148808 3169 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.155938 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:04.176350 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.158903 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.173503 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:04.151314 1892 catalog_manager.cc:5495] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
W20260924 00:22:04.165766 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 61608.5us
I20260924 00:22:04.178987 2036 leader_election.cc:398] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:04.179262 2036 leader_election.cc:401] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260924 00:22:04.142702 3187 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:04.083346 3311 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.154655 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:04.139972 1893 rpcz_store.cc:276] 0924 00:22:01.539125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.551071 (+ 11946us) service_pool.cc:227] Handling call
0924 00:22:02.507924 (+956853us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b5400002680. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d8403361 00007fb6d84d3a86 00007fb6d84c9e03 00007fb6d848ece7 00007fb6d848eaf4 00007fb6dab431f1 00007fb6dab43a16 00007fb6daa0dd7b 00007fb6dab114fe 00007fb6d8e22fb4 00007fb6d8e22f42
0924 00:22:04.128097 (+1620173us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:02.847342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:02.899190 (+ 51848us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:02.899235 (+ 45us) write_op.cc:435] Acquired schema lock
0924 00:22:02.899245 (+ 10us) tablet.cc:660] Decoding operations
0924 00:22:02.899669 (+ 424us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:02.899730 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:02.899754 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:02.899937 (+ 183us) tablet.cc:699] Row locks acquired
0924 00:22:02.899952 (+ 15us) write_op.cc:260] PREPARE: finished
0924 00:22:02.900103 (+ 151us) write_op.cc:270] Start()
0924 00:22:02.900221 (+ 118us) write_op.cc:276] Timestamp: P: 1790209322900056 usec, L: 0
0924 00:22:02.900240 (+ 19us) op_driver.cc:331] REPLICATION: starting
0924 00:22:02.901059 (+ 819us) log.cc:842] Serialized 530 byte log entry
0924 00:22:03.782074 (+881015us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b18002365c0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d84cc8a8 00007fb6d84cea78 00007fb6d84cea28 00007fb6d84ce9a8 00007fb6d84ce963 00007fb6d84cdc4c 00007fb6daac5206 00007fb6daabd808 00007fb6d4ac8170
0924 00:22:03.782153 (+ 79us) op_driver.cc:444] REPLICATION: finished
0924 00:22:03.964382 (+182229us) write_op.cc:301] APPLY: starting
0924 00:22:03.964470 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:03.964593 (+ 123us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:03.964613 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:03.964845 (+ 232us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:03.967762 (+ 2917us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:03.967856 (+ 94us) write_op.cc:312] APPLY: finished
0924 00:22:03.968095 (+ 239us) log.cc:842] Serialized 23 byte log entry
0924 00:22:04.050611 (+ 82516us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:04.054586 (+ 3975us) write_op.cc:454] Released schema lock
0924 00:22:04.079517 (+ 24931us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:04.096753 (+ 17236us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:04.099980 (+ 3227us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2721,"reported_tablets":3,"spinlock_wait_cycles":14824320,"child_traces":[["op",{"apply.queue_time_us":181942,"apply.run_cpu_time_us":42272,"apply.run_wall_time_us":135896,"num_ops":1,"prepare.queue_time_us":308250,"prepare.run_cpu_time_us":3050,"prepare.run_wall_time_us":90558,"replication_time_us":26311,"spinlock_wait_cycles":51539200,"thread_start_us":336572,"threads_started":3,"wal-append.queue_time_us":138972}]]}
I20260924 00:22:04.106837 3207 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:04.146380 3307 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.178135 2143 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:22:04.164729 3254 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.163285 3232 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260924 00:22:04.166116 3277 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260924 00:22:04.181257 3187 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:04.176059 3296 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:04.171271 3274 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:04.110934 3243 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:04.163416 3208 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:04.151611 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 67376.3us
I20260924 00:22:04.180544 3241 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.163932 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.171399 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.167518 3275 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.171722 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:04.161373 3199 consensus_queue.cc:237] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.257555 3294 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.262247 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:04.226926 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.216949 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:04.231376 3321 log.cc:924] Time spent T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260924 00:22:04.285768 3318 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:04.226432 3207 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:04.288487 3254 consensus_queue.cc:1046] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:04.289351 3254 raft_consensus.cc:3032] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:04.289686 3254 raft_consensus.cc:729] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:04.286837 3274 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:04.290505 3254 consensus_queue.cc:260] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.291574 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.291663 3254 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:04.292685 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.289529 3243 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:04.236900 3230 consensus_meta.cc:287] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.724s user 0.009s sys 0.000s
I20260924 00:22:04.171581 3292 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:04.300166 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 187787us
I20260924 00:22:04.300868 2613 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:04.303859 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:04.186863 3061 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
W20260924 00:22:04.245618 3326 log.cc:924] Time spent T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260924 00:22:04.309074 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:04.312388 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:04.236728 3320 log.cc:924] Time spent T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20260924 00:22:04.313601 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:04.193897 3249 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.257908 3250 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:04.250533 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.315521 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.246637 3228 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.247771 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.294838 3230 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.329754 2613 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:04.331029 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.226346 3311 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.301426 2144 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:04.306056 2138 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:04.310613 3243 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:04.317044 3273 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:04.344053 3273 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:04.344470 3273 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:04.347281 2138 raft_consensus.cc:1226] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:04.193167 3300 consensus_queue.cc:237] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.323100 3265 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:04.349083 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.332691 3215 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:04.325639 3250 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.350811 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.352099 3274 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.352458 3250 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.310573 3294 consensus_queue.cc:1046] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260924 00:22:04.314394 3195 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:04.342248 3296 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:04.343612 3187 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:04.348515 2138 raft_consensus.cc:1226] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:04.316679 3061 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:04.093219 2136 rpcz_store.cc:276] 0924 00:22:02.618164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.619219 (+ 1055us) service_pool.cc:227] Handling call
0924 00:22:02.648465 (+ 29246us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:02.685829 (+ 37364us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:02.688084 (+ 2255us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:02.808047 (+119963us) log.cc:842] Serialized 26 byte log entry
0924 00:22:03.292885 (+484838us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:03.320694 (+ 27809us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.320811 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:04.022402 (+701591us) raft_consensus.cc:1671] finished
0924 00:22:04.022439 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:04.038707 (+ 16268us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6469,"raft.queue_time_us":11699,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":8881,"thread_start_us":261806,"threads_started":1,"wal-append.queue_time_us":814654}
I20260924 00:22:04.306689 3207 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:03.813037 2068 rpcz_store.cc:276] 0924 00:22:02.688951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:02.839561 (+150610us) service_pool.cc:227] Handling call
0924 00:22:02.839931 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:02.840138 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 5
0924 00:22:03.275399 (+435261us) log.cc:842] Serialized 14 byte log entry
0924 00:22:03.766389 (+490990us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:03.766529 (+ 140us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:03.766658 (+ 129us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:03.767038 (+ 380us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:03.767393 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":830416,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":60504,"thread_start_us":892838,"threads_started":2}
I20260924 00:22:04.354646 1991 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:04.355141 2139 raft_consensus.cc:1226] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:04.433718 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.436610 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.435580 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.436878 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.427654 3230 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.434792 3061 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.447149 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.448475 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.437299 3233 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.428969 3308 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.434103 2142 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:04.349103 3265 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:04.439116 2068 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:04.523017 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.522876 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.428563 2144 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:04.406527 3168 raft_consensus.cc:3032] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:04.523825 2068 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:04.353508 3243 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:04.527184 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 81283us
I20260924 00:22:04.530433 2036 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:04.538237 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:04.538983 2036 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:04.539075 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.539717 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:04.540231 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:04.540244 1996 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:04.542184 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.543529 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:04.544040 2142 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:04.531253 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:04.525666 3265 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:04.548748 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.549790 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.558671 3311 consensus_queue.cc:1046] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:04.559481 3172 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:04.559866 3172 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:04.561313 3274 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.529080 3195 raft_consensus.cc:686] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:04.541430 3243 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:04.529336 3168 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:04.526710 2069 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:04.541533 3241 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:04.592150 3206 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:04.571205 1996 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:04.573810 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:04.544471 3333 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:04.618609 3333 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:04.619100 3333 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:04.621301 3040 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:04.546250 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.623909 3207 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:04.560702 3172 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.609443 3195 consensus_queue.cc:237] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.556310 3295 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:04.554975 3249 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:04.615150 2110 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:04.631177 3323 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:04.631750 3323 raft_consensus.cc:686] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:04.632617 3323 consensus_queue.cc:237] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.624724 3296 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:04.637393 3168 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.638793 3168 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:04.617282 3302 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.944s user 0.000s sys 0.004s
I20260924 00:22:04.617614 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:04.618747 3343 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260924 00:22:04.540989 2064 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
W20260924 00:22:04.641593 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 101389us
I20260924 00:22:04.625530 3169 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 00:22:04.691789 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 65556us
I20260924 00:22:04.691972 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:04.692962 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:04.629644 3274 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260924 00:22:04.691368 3307 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:04.753098 3346 log.cc:924] Time spent T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260924 00:22:04.752704 3307 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:04.754927 3307 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:04.694330 3310 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:04.705931 3241 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:04.829603 3241 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:04.829984 3241 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:04.813711 3187 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:04.815456 3170 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:04.833424 3187 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.833524 3170 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.835566 3170 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.689966 1987 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:22:04.825644 3243 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:04.699303 3207 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.103s
I20260924 00:22:04.732321 3294 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:04.741254 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:04.820353 3333 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.852473 2642 raft_consensus.cc:1226] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:04.864147 3243 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.904137 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.905930 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:04.907333 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:04.868211 3061 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.942875 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:04.860013 3273 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.951407 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.980111 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:04.981002 2068 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:04.953568 3040 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:04.987499 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:04.989616 3040 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:04.990407 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.992018 3040 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.992848 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:04.993929 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:04.997095 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:04.997777 2070 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:05.000389 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.002038 2063 raft_consensus.cc:1226] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260924 00:22:04.943184 3308 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:05.000980 2067 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:05.008721 2067 raft_consensus.cc:1261] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260924 00:22:05.005614 1995 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:05.010484 2067 pending_rounds.cc:76] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 1
I20260924 00:22:05.003414 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.011272 2063 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
W20260924 00:22:05.005510 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 308526us
I20260924 00:22:05.015760 2614 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:05.016034 3333 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:04.960209 3187 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.001742 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.003180 3249 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:05.021126 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 883) took 1040 ms. Trace:
I20260924 00:22:05.027781 2065 rpcz_store.cc:276] 0924 00:22:03.980207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.982829 (+ 2622us) service_pool.cc:227] Handling call
0924 00:22:04.027122 (+ 44293us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:04.063387 (+ 36265us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:04.063416 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:04.063466 (+ 50us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:04.063499 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.020853 (+957354us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x7b5400427cd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:05.020908 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:05.021061 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82971904}
I20260924 00:22:05.023631 1994 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:05.021750 2070 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:05.030755 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:05.010802 2067 pending_rounds.cc:97] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260924 00:22:05.040755 2067 raft_consensus.cc:2866] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:05.040705 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:05.016769 3308 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.040391 3300 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:05.031594 2644 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:05.169345 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.162370 3338 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:05.129297 3169 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:05.133029 3243 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 00:22:05.174256 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 133611us
I20260924 00:22:05.175122 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:05.175402 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.033252 3249 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.180948 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.182886 1994 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:05.186520 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:05.187547 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:05.162755 3348 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:05.151253 3230 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260924 00:22:05.165423 3170 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:05.172631 3338 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:05.204981 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.171802 3300 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.190563 3169 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:05.199786 3295 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:05.250305 2145 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:05.209569 3338 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:05.252399 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:05.258165 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 104544us
I20260924 00:22:05.263366 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.272509 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.274587 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.276007 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.278514 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.251299 3273 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.200688 3170 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:05.253721 3295 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:05.198308 3348 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:05.259315 3364 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:05.292493 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.273964 3274 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:05.290949 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 1356) took 1344 ms. Trace:
W20260924 00:22:05.290940 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 96772.6us
I20260924 00:22:05.250278 3169 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:05.299916 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.299253 2141 rpcz_store.cc:276] 0924 00:22:03.946454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.949018 (+ 2564us) service_pool.cc:227] Handling call
0924 00:22:03.949320 (+ 302us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:04.166341 (+217021us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:04.166376 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:04.166792 (+ 416us) log.cc:842] Serialized 26 byte log entry
0924 00:22:04.314930 (+148138us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:04.314970 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.275091 (+960121us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b540037f0d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:05.275140 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:05.275179 (+ 39us) raft_consensus.cc:1671] finished
0924 00:22:05.275191 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:05.290888 (+ 15697us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5621,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":17144,"spinlock_wait_cycles":485944704,"thread_start_us":143951,"threads_started":1,"wal-append.queue_time_us":149337}
I20260924 00:22:05.304077 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.300285 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 59) took 1341 ms. Trace:
I20260924 00:22:05.299204 3274 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:05.291563 3295 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:05.279286 2145 raft_consensus.cc:2362] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3. Re-sending same reply.
I20260924 00:22:05.294217 3364 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:05.305501 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:05.333945 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.320765 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.339588 1960 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:05.307845 3274 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:05.306308 3347 log.cc:924] Time spent T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260924 00:22:05.307089 2137 rpcz_store.cc:276] 0924 00:22:03.959004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.963344 (+ 4340us) service_pool.cc:227] Handling call
0924 00:22:03.963683 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:04.157551 (+193868us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:04.164312 (+ 6761us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:04.166807 (+ 2495us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:04.171790 (+ 4983us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.286954 (+1115164us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b540021be54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:05.288829 (+ 1875us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:05.294045 (+ 5216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":418414848}
I20260924 00:22:05.351707 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.311245 3308 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:05.329556 3350 log.cc:924] Time spent T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260924 00:22:05.353068 3348 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
W20260924 00:22:05.306070 3355 log.cc:924] Time spent T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260924 00:22:05.353971 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:05.737902 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 386266us
I20260924 00:22:05.738487 3187 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:05.739010 3187 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:05.324033 3364 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:05.737922 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.739877 3187 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.737277 3172 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:05.330785 3265 consensus_meta.cc:287] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.764s user 0.010s sys 0.000s
I20260924 00:22:05.534144 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1028) took 1012 ms. Trace:
I20260924 00:22:05.743752 1995 rpcz_store.cc:276] 0924 00:22:04.521593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.524407 (+ 2814us) service_pool.cc:227] Handling call
0924 00:22:04.524689 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:05.013363 (+488674us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.533860 (+520497us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b5400073fd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:05.534059 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105535744}
I20260924 00:22:05.718204 2143 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 966) took 1686 ms. Trace:
I20260924 00:22:05.745586 2143 rpcz_store.cc:276] 0924 00:22:04.032289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.290847 (+258558us) service_pool.cc:227] Handling call
0924 00:22:04.340429 (+ 49582us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:04.340653 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:04.340689 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:04.340742 (+ 53us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:04.340783 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:04.340853 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:05.717916 (+1377063us) spinlock_profiling.cc:243] Waited 49.3 ms on lock 0x7b540021c350. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abba09 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:05.718125 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113384832}
I20260924 00:22:05.354539 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.459192 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.749312 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.523038 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1026) took 1352 ms. Trace:
I20260924 00:22:05.751135 1991 rpcz_store.cc:276] 0924 00:22:04.170451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.293824 (+123373us) service_pool.cc:227] Handling call
0924 00:22:04.294073 (+ 249us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:04.439437 (+145364us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.522738 (+1083301us) spinlock_profiling.cc:243] Waited 60.3 ms on lock 0x7b540004bad4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:05.522968 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138766592}
I20260924 00:22:05.521340 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.759660 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 1036) took 1226 ms. Trace:
I20260924 00:22:05.412716 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.764835 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.311802 3294 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:05.760098 1992 rpcz_store.cc:276] 0924 00:22:04.533738 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.533905 (+ 167us) service_pool.cc:227] Handling call
0924 00:22:05.521163 (+987258us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:05.759588 (+238425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13126912}
W20260924 00:22:05.564786 2035 leader_election.cc:334] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 1.500s (SENT)
W20260924 00:22:05.437966 3241 consensus_meta.cc:287] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.608s user 0.010s sys 0.000s
I20260924 00:22:05.774144 3294 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.776073 2035 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:05.428251 3354 log.cc:924] Time spent T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20260924 00:22:05.313222 3169 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:05.652287 3199 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.013s sys 0.013s
I20260924 00:22:05.306770 3170 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:05.331347 3249 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.772361 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.816596 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.345103 3040 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:05.773797 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.774492 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.662160 3273 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:05.428876 3363 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20260924 00:22:05.777279 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:05.766848 2036 leader_election.cc:334] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.552s (SENT)
W20260924 00:22:05.702009 3345 log.cc:924] Time spent T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 1.076s user 0.010s sys 0.006s
I20260924 00:22:05.742274 1988 raft_consensus.cc:2379] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:05.782598 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.769011 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:05.787065 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 222314us
W20260924 00:22:05.851001 2035 leader_election.cc:334] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 1.575s (SENT)
I20260924 00:22:05.789654 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.518623 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.437939 3367 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:05.824429 3040 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:05.519717 3210 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:05.814167 1992 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260924 00:22:05.476107 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 459194us
W20260924 00:22:05.820324 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 80607.5us
I20260924 00:22:05.820982 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:05.791793 3199 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s user 0.025s sys 0.014s
I20260924 00:22:05.864683 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1032) took 1336 ms. Trace:
I20260924 00:22:05.822259 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 1349) took 1890 ms. Trace:
I20260924 00:22:05.431259 3338 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:05.822655 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 64110.6us
I20260924 00:22:05.848299 3253 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:05.873466 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.825660 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.874961 2140 raft_consensus.cc:2379] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:05.517290 3369 consensus_queue.cc:1035] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.324s
I20260924 00:22:05.878003 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.768491 3265 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.878695 2065 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:05.880214 3265 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.790717 3308 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:05.880908 3308 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:05.882248 3375 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:05.882611 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47766 (request call id 894) took 1438 ms. Trace:
I20260924 00:22:05.882812 3375 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.883051 2064 rpcz_store.cc:276] 0924 00:22:04.443946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.448169 (+ 4223us) service_pool.cc:227] Handling call
0924 00:22:05.882438 (+1434269us) spinlock_profiling.cc:243] Waited 8.98 ms on lock 0x7b54003f93d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:05.882552 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20661120}
W20260924 00:22:05.774544 3168 consensus_meta.cc:287] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 1.133s user 0.008s sys 0.002s
I20260924 00:22:05.788957 3234 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:05.776301 3241 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:05.885723 1989 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:05.887004 3241 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:05.887637 2071 raft_consensus.cc:1226] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:05.889514 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1038) took 1353 ms. Trace:
I20260924 00:22:05.890043 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:05.890993 2063 raft_consensus.cc:2379] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:05.889906 1990 rpcz_store.cc:276] 0924 00:22:04.535790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.623169 (+ 87379us) service_pool.cc:227] Handling call
0924 00:22:04.623619 (+ 450us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:04.643593 (+ 19974us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:04.647744 (+ 4151us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:04.821102 (+173358us) log.cc:842] Serialized 26 byte log entry
0924 00:22:05.156808 (+335706us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:05.163149 (+ 6341us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.172243 (+ 9094us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:05.889236 (+716993us) raft_consensus.cc:1671] finished
0924 00:22:05.889261 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:05.889435 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41916,"raft.queue_time_us":18040,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":44668,"thread_start_us":302001,"threads_started":1,"wal-append.queue_time_us":378566}
W20260924 00:22:05.891969 2109 connection.cc:232] RPC call timeout handler was delayed by -2.5900345770000968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
I20260924 00:22:05.892374 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.892921 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.827452 3273 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:05.893522 2143 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:05.894767 3273 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.894722 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.895413 2647 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:05.896293 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.896636 3338 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.897517 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.898125 2641 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:05.899276 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.899506 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.899899 2067 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:05.894214 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.901095 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.851747 2035 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:05.865253 2139 rpcz_store.cc:276] 0924 00:22:03.932389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.932613 (+ 224us) service_pool.cc:227] Handling call
0924 00:22:03.932939 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:04.445513 (+512574us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.822005 (+1376492us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b540025e754. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:05.822199 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":970472832}
I20260924 00:22:05.909090 2069 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260924 00:22:05.910259 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 59288.5us
I20260924 00:22:05.911831 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.866139 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.913870 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.915062 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.865118 1993 rpcz_store.cc:276] 0924 00:22:04.528658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.528805 (+ 147us) service_pool.cc:227] Handling call
0924 00:22:04.529103 (+ 298us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:04.529315 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:04.529340 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:04.529738 (+ 398us) log.cc:842] Serialized 26 byte log entry
0924 00:22:04.728407 (+198669us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:04.728458 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:04.728555 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:05.864495 (+1135940us) raft_consensus.cc:1671] finished
0924 00:22:05.864513 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:05.864631 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":164761,"threads_started":1,"wal-append.queue_time_us":479755}
I20260924 00:22:05.916826 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:05.917474 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:05.869168 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.871831 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:05.872308 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.824092 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:05.918865 2068 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:05.918805 2070 raft_consensus.cc:2362] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4. Re-sending same reply.
I20260924 00:22:05.919323 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.812820 3274 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:05.875916 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.883160 3308 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:05.867447 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 1034) took 1326 ms. Trace:
I20260924 00:22:05.857578 3367 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:05.924628 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:05.945343 3367 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:05.922590 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:05.924535 1993 raft_consensus.cc:2379] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
W20260924 00:22:05.928221 3295 consensus_meta.cc:287] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.605s user 0.010s sys 0.000s
I20260924 00:22:05.885434 1987 raft_consensus.cc:2396] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:05.975737 2036 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:05.956784 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:05.983108 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:06.062520 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 86693us
I20260924 00:22:06.063964 2064 raft_consensus.cc:2379] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:05.942940 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.968197 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.067219 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.068375 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.952489 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.069824 2065 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:06.071132 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.071913 2065 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:05.966403 2066 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:05.934083 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.922760 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.852263 1994 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:05.954378 2069 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260924 00:22:05.981994 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 125206us
I20260924 00:22:06.069038 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.935146 1988 rpcz_store.cc:276] 0924 00:22:04.532003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.537694 (+ 5691us) service_pool.cc:227] Handling call
0924 00:22:05.737808 (+1200114us) spinlock_profiling.cc:243] Waited 7.07 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5cbce 000000000034aea9 00007fb6d9988a30 00007fb6d9a4e7b4 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:05.858756 (+120948us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16260864}
I20260924 00:22:06.088261 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 996) took 1101 ms. Trace:
I20260924 00:22:05.940017 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.076925 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:05.947948 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.075246 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.091012 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.092248 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.092506 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.078549 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:06.056422 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 95428.4us
I20260924 00:22:06.094826 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.096083 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.080528 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:06.082214 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 189507us
W20260924 00:22:06.059703 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 137040us
I20260924 00:22:06.099764 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.073124 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.105279 2065 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:05.863976 3249 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:06.146391 2063 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:06.088718 2136 rpcz_store.cc:276] 0924 00:22:04.986510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.989235 (+ 2725us) service_pool.cc:227] Handling call
0924 00:22:04.989553 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:05.176852 (+187299us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:05.176891 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:05.248743 (+ 71852us) log.cc:842] Serialized 26 byte log entry
0924 00:22:05.498652 (+249909us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:05.498742 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:05.498873 (+ 131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:06.018672 (+519799us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b24002692b8. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d19eccf4 00007fb6d19eadfb 00007fb6d1a79559 00007fb6d1a791e5 00007fb6d1a78740 00007fb6d476b578 00007fb6d476bcd8 00007fb6d47436e8 00007fb6d473eaea 00007fb6d473df36
0924 00:22:06.087619 (+ 68947us) raft_consensus.cc:1671] finished
0924 00:22:06.087665 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:06.088034 (+ 369us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10574,"raft.queue_time_us":11176,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":99,"spinlock_wait_cycles":44284288,"thread_start_us":348300,"threads_started":2,"wal-append.queue_time_us":588744}
I20260924 00:22:06.102069 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1044) took 1077 ms. Trace:
I20260924 00:22:06.149809 2067 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:06.149562 1993 rpcz_store.cc:276] 0924 00:22:05.007088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:06.063652 (+1056564us) service_pool.cc:227] Handling call
0924 00:22:06.064466 (+ 814us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:06.064948 (+ 482us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:06.065002 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:06.065074 (+ 72us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:06.084364 (+ 19290us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.084485 (+ 121us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:06.084634 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:06.151640 1993 raft_consensus.cc:1261] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:05.895426 2614 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.006392 3367 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:06.155007 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 259598us
I20260924 00:22:06.155783 1991 raft_consensus.cc:2379] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:05.939968 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.935948 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:05.957674 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:06.159376 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:06.160054 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1388) took 1214 ms. Trace:
I20260924 00:22:06.160521 1991 rpcz_store.cc:276] 0924 00:22:04.945957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:05.773490 (+827533us) service_pool.cc:227] Handling call
0924 00:22:06.159870 (+386380us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b54000758d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:06.160007 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":770173696}
I20260924 00:22:06.161108 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:06.163985 3380 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:06.164441 3380 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:06.080215 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.165561 3376 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:06.165983 3376 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:06.166293 3376 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.166553 2069 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:06.166867 3367 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.864109 3040 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.170764 3312 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.171211 3312 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:06.171612 3312 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.113845 3061 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:06.173815 3061 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:06.174240 3061 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.114835 3207 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:06.177518 3207 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.179906 3207 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.115012 3170 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:06.180425 3170 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:06.180790 3170 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.114881 3066 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:06.185510 3066 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.187842 3066 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.975078 3274 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:05.927870 3338 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.188935 3338 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:06.189558 3338 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.189584 2640 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:06.190104 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.192276 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:06.030983 3379 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.142s user 0.000s sys 0.002s
I20260924 00:22:06.193176 2143 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:22:06.195102 2613 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:05.921257 3199 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:06.197633 3271 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.197095 3199 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.198114 3271 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:06.198553 3271 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:22:06.198884 1987 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1381) took 1298 ms (client timeout 1552 ms). Trace:
W20260924 00:22:06.199354 1987 rpcz_store.cc:269] 0924 00:22:04.900580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.907054 (+ 6474us) service_pool.cc:227] Handling call
0924 00:22:06.198527 (+1291473us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b5400159d54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:06.198695 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2246794624}
I20260924 00:22:05.900161 3210 consensus_queue.cc:1046] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260924 00:22:05.951700 3234 consensus_queue.cc:1046] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260924 00:22:05.929769 3241 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.077908 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1387) took 1132 ms. Trace:
I20260924 00:22:06.242763 1994 rpcz_store.cc:276] 0924 00:22:04.945623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:05.773181 (+827558us) service_pool.cc:227] Handling call
0924 00:22:06.077672 (+304491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:06.243351 3234 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:06.243963 3234 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:05.920976 3253 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.223321 3368 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:06.245941 3368 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.247256 3253 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.957577 3294 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.964780 3295 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.248656 3368 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.254987 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:05.967767 3375 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:05.971601 3377 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:06.089185 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.128361 3296 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260924 00:22:06.026863 3371 log.cc:924] Time spent T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.173s user 0.002s sys 0.000s
I20260924 00:22:06.132524 3311 consensus_queue.cc:1046] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.267765 3311 raft_consensus.cc:3032] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:06.267207 3377 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.268272 3311 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:06.269587 3311 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.271713 3311 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.133158 2642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 268) took 1690 ms. Trace:
I20260924 00:22:06.135617 3348 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.273377 2642 rpcz_store.cc:276] 0924 00:22:04.442725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.447818 (+ 5093us) service_pool.cc:227] Handling call
0924 00:22:04.522491 (+ 74673us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:04.872319 (+349828us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.132429 (+1260110us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b540061e4d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:06.132831 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":749028736}
I20260924 00:22:06.138542 3323 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.275710 3323 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:06.006309 3300 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.140296 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1029) took 1618 ms. Trace:
I20260924 00:22:06.141091 3360 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.146171 3249 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.147135 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.004437 3364 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.164896 3380 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.151602 2067 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260924 00:22:06.281250 2065 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:06.086499 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:06.206293 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 107936us
I20260924 00:22:06.188666 3312 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.205822 3199 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:06.212442 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 115822us
W20260924 00:22:06.222649 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 115646us
I20260924 00:22:06.223174 2063 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:06.096869 2070 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.249377 1987 raft_consensus.cc:1226] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:06.285044 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.250638 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 1392) took 1300 ms. Trace:
I20260924 00:22:06.251029 3295 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.252723 3172 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:06.285879 2065 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:06.252060 3234 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.253108 1990 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:06.223428 2144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 997) took 1226 ms. Trace:
I20260924 00:22:06.248713 1992 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:06.297732 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.278219 1989 rpcz_store.cc:276] 0924 00:22:04.522084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.525154 (+ 3070us) service_pool.cc:227] Handling call
0924 00:22:05.541452 (+1016298us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:05.541661 (+ 209us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:05.886181 (+344520us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.139708 (+253527us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b540004c4d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:06.140087 (+ 379us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":797099648}
I20260924 00:22:06.257670 3061 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.304550 2069 raft_consensus.cc:1226] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:06.307204 2069 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:06.307057 3360 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:06.309949 2614 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.259182 3040 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.312644 1996 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:06.313346 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.313926 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1037) took 1778 ms. Trace:
I20260924 00:22:06.314260 1996 rpcz_store.cc:276] 0924 00:22:04.535058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.613072 (+ 78014us) service_pool.cc:227] Handling call
0924 00:22:05.826625 (+1213553us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:05.826818 (+ 193us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:06.313004 (+486186us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.313648 (+ 644us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54000753d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:06.313877 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1142172416}
I20260924 00:22:06.315527 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.290190 3364 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.318789 3040 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.276233 3323 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.236502 3170 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.275699 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.321564 3323 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.323118 3323 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.323655 3234 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:06.286638 2138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57748 (request call id 986) took 1746 ms (client timeout 1676 ms). Trace:
I20260924 00:22:06.325647 3372 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.288205 1988 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:06.290150 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 103) took 1591 ms. Trace:
I20260924 00:22:06.290923 3312 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.290400 3172 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.292246 3377 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.285916 1995 rpcz_store.cc:276] 0924 00:22:04.949745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:05.853569 (+903824us) service_pool.cc:227] Handling call
0924 00:22:05.853819 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:05.854003 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 2
0924 00:22:05.962673 (+108670us) log.cc:842] Serialized 14 byte log entry
0924 00:22:06.249986 (+287313us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:06.250083 (+ 97us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:06.250176 (+ 93us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.250286 (+ 110us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:06.250445 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":117322,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":4778,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":116,"thread_start_us":351648,"threads_started":2,"wal-append.queue_time_us":319026}
I20260924 00:22:06.332327 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:06.326779 2071 rpcz_store.cc:276] 0924 00:22:04.698142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.698247 (+ 105us) service_pool.cc:227] Handling call
0924 00:22:04.698464 (+ 217us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:05.888005 (+1189541us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.287107 (+399102us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54004281d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:06.289959 (+ 2852us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2734449536}
I20260924 00:22:06.294402 3249 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.332819 3172 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.288290 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.333559 2063 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.333745 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:06.334676 1995 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:06.296820 2144 rpcz_store.cc:276] 0924 00:22:04.986926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.988861 (+ 1935us) service_pool.cc:227] Handling call
0924 00:22:04.989205 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:05.275590 (+286385us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:05.284410 (+ 8820us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:05.308316 (+ 23906us) log.cc:842] Serialized 26 byte log entry
0924 00:22:05.827208 (+518892us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:05.983608 (+156400us) log.cc:842] Serialized 14 byte log entry
0924 00:22:06.030636 (+ 47028us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b24002692b8. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d19eccf4 00007fb6d19eadfb 00007fb6d1a79559 00007fb6d1a791e5 00007fb6d1a7882d 00007fb6d472746e 00007fb6d473ec0c 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551
0924 00:22:06.091936 (+ 61300us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.092095 (+ 159us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:06.200182 (+108087us) raft_consensus.cc:1671] finished
0924 00:22:06.202894 (+ 2712us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:06.212922 (+ 10028us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8985,"prepare.queue_time_us":135939,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":163422,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":400,"spinlock_wait_cycles":27225984,"thread_start_us":801613,"threads_started":3,"wal-append.queue_time_us":580443}
I20260924 00:22:06.301553 1989 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:06.337263 1995 raft_consensus.cc:1226] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:06.301976 2064 raft_consensus.cc:2396] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:06.302541 3386 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:06.338626 3386 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.339066 3391 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:06.339653 3391 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.340031 3386 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.260532 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33246 (request call id 214) took 4348 ms. Trace:
W20260924 00:22:06.325812 2138 rpcz_store.cc:269] 0924 00:22:04.540459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:04.542287 (+ 1828us) service_pool.cc:227] Handling call
0924 00:22:06.146791 (+1604504us) spinlock_profiling.cc:243] Waited 7.88 ms on lock 0x7b5800004298. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:06.286540 (+139749us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18121984}
I20260924 00:22:06.341260 1989 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:06.342303 3170 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.343165 3392 consensus_queue.cc:1046] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:06.344462 3386 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.343286 2063 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:06.344820 3392 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:06.349421 3392 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.342020 1892 rpcz_store.cc:276] 0924 00:22:01.912440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:01.912639 (+ 199us) service_pool.cc:227] Handling call
0924 00:22:06.260280 (+4347641us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:04.438558 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:04.696692 (+258134us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:04.700002 (+ 3310us) write_op.cc:435] Acquired schema lock
0924 00:22:04.702671 (+ 2669us) tablet.cc:660] Decoding operations
0924 00:22:04.761265 (+ 58594us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:04.761350 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:04.761380 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0924 00:22:04.761706 (+ 326us) tablet.cc:699] Row locks acquired
0924 00:22:04.761723 (+ 17us) write_op.cc:260] PREPARE: finished
0924 00:22:04.761971 (+ 248us) write_op.cc:270] Start()
0924 00:22:04.762108 (+ 137us) write_op.cc:276] Timestamp: P: 1790209324761895 usec, L: 0
0924 00:22:04.762137 (+ 29us) op_driver.cc:331] REPLICATION: starting
0924 00:22:04.800666 (+ 38529us) log.cc:842] Serialized 917 byte log entry
0924 00:22:05.006450 (+205784us) op_driver.cc:444] REPLICATION: finished
0924 00:22:05.977946 (+971496us) write_op.cc:301] APPLY: starting
0924 00:22:05.978052 (+ 106us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:05.978202 (+ 150us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:05.978365 (+ 163us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:05.978779 (+ 414us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:05.978997 (+ 218us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 00:22:05.979063 (+ 66us) write_op.cc:312] APPLY: finished
0924 00:22:05.979298 (+ 235us) log.cc:842] Serialized 29 byte log entry
0924 00:22:06.019449 (+ 40151us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:06.019574 (+ 125us) write_op.cc:454] Released schema lock
0924 00:22:06.019625 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:06.019860 (+ 235us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:06.019902 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":951830,"apply.run_cpu_time_us":24405,"apply.run_wall_time_us":42272,"num_ops":2,"prepare.queue_time_us":240553,"prepare.run_cpu_time_us":4460,"prepare.run_wall_time_us":362416,"replication_time_us":242449,"thread_start_us":715051,"threads_started":3,"wal-append.queue_time_us":46112}]]}
I20260924 00:22:06.350844 3392 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.352317 3271 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.352890 3392 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.360143 2640 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:06.363590 2070 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:06.366741 3256 consensus_queue.cc:1046] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:06.369760 3353 raft_consensus.cc:3032] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:06.370003 2068 raft_consensus.cc:1226] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:06.370260 3353 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:06.372617 2035 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.373574 2109 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:06.375275 3199 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.375371 2109 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.375841 3199 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:06.316944 3338 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.376328 3199 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.376381 3382 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:06.377064 2109 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.357259 3376 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.379035 3382 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.380254 2109 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:06.360852 3061 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.385087 1996 raft_consensus.cc:2379] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:06.394450 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1043) took 1388 ms. Trace:
I20260924 00:22:06.395035 1990 rpcz_store.cc:276] 0924 00:22:05.006489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:05.891691 (+885202us) service_pool.cc:227] Handling call
0924 00:22:05.891967 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:06.296427 (+404460us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:06.394141 (+ 97714us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b5400159854. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:06.394387 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":829611264}
I20260924 00:22:06.396565 3338 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.396692 3382 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.397294 3382 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:06.397706 3382 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.403407 3353 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.405004 3353 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.410826 1891 catalog_manager.cc:5495] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:22:06.412436 2109 consensus_peers.cc:487] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2ef9fa9e5dd48218a5c243de0c0e49a. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:06.411963 3382 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.416512 2109 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:06.420272 3359 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.420842 3359 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:06.421762 3359 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.424578 3376 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.386945 3172 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.425325 3332 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:06.426076 3332 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.420868 3380 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.415202 1891 catalog_manager.cc:5495] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 2, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 2 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:22:06.443109 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 74644.9us
I20260924 00:22:06.444554 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.445012 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.445387 3061 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:06.445838 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.437227 3376 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:06.449235 1987 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:06.450905 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.468843 3391 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.469151 2035 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.469517 3391 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.470233 3391 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.470538 2035 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.470755 3376 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.471851 2035 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.473368 3061 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.474133 3061 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.474274 3271 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.454681 3380 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.447309 3338 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:06.448618 3332 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.457571 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.458989 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.453585 1891 catalog_manager.cc:5495] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 2, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 2 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:06.460292 1960 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.461009 2136 raft_consensus.cc:1261] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:06.461170 3372 consensus_queue.cc:1046] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:06.464905 2614 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.465055 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.467245 2137 raft_consensus.cc:1226] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:06.480906 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.481901 2035 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.498716 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.499374 3391 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.499912 2142 raft_consensus.cc:2379] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:06.492293 2109 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:06.501053 2035 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.487883 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.502377 3372 raft_consensus.cc:3032] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:06.505486 3380 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.506916 2144 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:06.507169 3391 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.508682 3332 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.509246 3372 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:06.512902 3372 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:06.513899 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 53727.6us
I20260924 00:22:06.511389 3332 consensus_queue.cc:1046] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260924 00:22:06.515878 3332 raft_consensus.cc:3032] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:06.516127 3377 consensus_queue.cc:1046] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:06.516398 3332 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:06.516503 3040 consensus_queue.cc:1046] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:06.517707 3040 raft_consensus.cc:3032] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:06.517823 3380 raft_consensus.cc:3032] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:06.518069 3040 raft_consensus.cc:729] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:06.518216 3380 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:06.519106 3372 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.519080 3040 consensus_queue.cc:260] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.509164 2035 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.492431 3199 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.506367 2063 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:22:06.520573 3040 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.473061 3376 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.526597 2613 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.530068 3393 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.530500 3393 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:06.530890 3393 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.530901 2142 raft_consensus.cc:1226] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:06.533659 2035 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.519117 3380 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.535038 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.535063 3380 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.538446 3312 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:06.538851 3312 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:06.539208 3312 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.542687 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.545099 2138 raft_consensus.cc:1226] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:06.545414 3332 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.545764 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.523665 3338 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.535054 2109 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.549790 3210 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.557293 3338 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.558811 3312 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.560834 3391 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:06.560113 2139 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:06.562253 2035 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.562685 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:06.560707 3393 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.563264 3391 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.522887 3382 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.564888 3393 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:06.565357 3199 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.565826 3312 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.567071 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.569015 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:06.569104 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:06.570531 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.571067 3332 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.571139 2063 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.527267 3234 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.577960 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.578890 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.579707 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.580125 2110 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:06.580382 1992 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.580492 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.578917 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.583993 2138 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:06.581261 3386 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:06.583187 3392 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:06.585951 3359 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.587502 2143 raft_consensus.cc:2379] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:06.585889 3392 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:06.588327 3392 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.585686 3386 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:06.589414 3386 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.590015 2142 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:06.590605 2613 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.593493 3271 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.593914 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:06.594228 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.595422 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:06.591171 2144 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:06.585606 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.597702 3199 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.599800 3199 consensus_queue.cc:1046] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.601221 3210 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:06.601558 3323 raft_consensus.cc:3032] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:06.601907 3323 raft_consensus.cc:729] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.602604 3210 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:06.603045 3210 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.602691 3323 consensus_queue.cc:260] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.604271 3323 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:06.603897 3210 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.605012 3210 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.607105 2109 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.608058 3360 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.611562 3360 consensus_queue.cc:1046] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:06.612825 3308 raft_consensus.cc:3032] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:06.613240 3308 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.614269 3308 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:06.617545 2035 leader_election.cc:334] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 1.805s (SENT)
I20260924 00:22:06.618453 2035 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.617787 3308 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.608878 2109 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.620414 3234 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.595991 3386 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.597898 2063 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:06.625074 2613 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.626713 1992 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:06.627408 3392 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.628364 2139 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:06.628649 3360 consensus_queue.cc:1046] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20260924 00:22:06.629192 3392 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:06.628259 3271 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:06.629832 3271 raft_consensus.cc:686] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:06.630932 3271 consensus_queue.cc:237] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.631958 2109 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.632368 3308 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.635741 3308 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.643126 3392 raft_consensus.cc:3032] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:06.643586 3392 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.644505 3392 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.645839 3392 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.649593 2109 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.650625 3210 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:06.651079 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.657232 1995 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:06.657493 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:06.659019 3040 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:06.641964 3359 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:06.659792 3040 raft_consensus.cc:686] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:06.658957 2110 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:06.660981 3040 consensus_queue.cc:237] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.662325 3360 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:06.665741 3360 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:06.666255 3360 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.668054 2109 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.668648 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.672484 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.681669 3210 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:06.675559 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:06.683264 1992 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:06.666421 3199 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:06.704931 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.709043 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:06.713939 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 46779.6us
I20260924 00:22:06.717067 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.704953 3376 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.717350 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.722278 1995 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:06.723799 2035 leader_election.cc:334] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 1.587s (SENT)
I20260924 00:22:06.724315 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.725871 3311 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.717042 2109 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.731760 2109 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.732705 3210 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.737296 3210 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.739065 2109 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.739995 3210 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:06.739920 3360 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.741576 2109 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.730520 1995 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:06.735965 1987 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:06.743429 3210 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.747589 2110 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:06.746670 3386 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:06.753244 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 84685.9us
I20260924 00:22:06.756567 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:06.757494 3376 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:06.757681 3199 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:06.758316 3199 raft_consensus.cc:686] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.758227 3376 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:06.759208 3376 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.759295 3199 consensus_queue.cc:237] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.756631 1893 catalog_manager.cc:5495] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 2 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260924 00:22:06.766556 2035 leader_election.cc:334] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 2.008s (SENT)
I20260924 00:22:06.767123 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.768329 3391 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:06.770803 2110 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:06.778646 3210 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
W20260924 00:22:06.778945 2035 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 2.086s (SENT) [suppressed 19 similar messages]
I20260924 00:22:06.791038 1960 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:06.802729 3040 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:06.806461 3040 raft_consensus.cc:686] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:06.794749 3210 raft_consensus.cc:686] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.798547 3323 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260924 00:22:06.817806 3040 consensus_queue.cc:237] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.818512 2109 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.820782 3359 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.820957 3408 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.821321 3392 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.809774 3210 consensus_queue.cc:237] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:06.822041 3392 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:06.825762 3360 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:06.826326 3380 consensus_queue.cc:1046] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.827664 3253 raft_consensus.cc:3032] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:06.828109 3253 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:06.830432 3253 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:06.839177 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 60257.9us
I20260924 00:22:06.858578 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:06.860936 3376 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.863416 3376 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.860661 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:06.864365 3391 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:06.864856 3391 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:06.865633 3391 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.870313 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.871346 3414 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:06.874949 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.875967 2139 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:06.877811 3386 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:06.878194 2109 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:06.877528 3393 consensus_queue.cc:1046] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260924 00:22:06.880185 3253 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.882609 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.884258 3386 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:06.884418 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.886391 2109 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:06.886771 3323 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:06.887208 3323 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.888185 3323 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.889307 3360 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:06.889856 3360 raft_consensus.cc:686] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:06.890749 3360 consensus_queue.cc:237] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.894703 3323 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.901268 2139 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:06.917750 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.917685 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.917709 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.919611 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.920250 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.920806 2065 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:06.923049 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.922199 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.926008 2139 raft_consensus.cc:2396] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:06.939625 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.940495 2063 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.942317 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.946434 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.946910 2070 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.948864 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.933949 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:06.939020 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.951756 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:06.918679 3348 consensus_queue.cc:1046] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260924 00:22:06.952780 2035 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:06.957446 3311 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:06.957653 3348 raft_consensus.cc:3032] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:06.958056 3348 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:06.959862 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:06.960649 2641 raft_consensus.cc:2379] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 2: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:06.957862 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:06.964030 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.964646 2142 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
W20260924 00:22:06.954885 3390 log.cc:924] Time spent T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260924 00:22:06.967057 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:06.967075 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:06.970115 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.958388 2064 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:06.973081 2063 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:06.972393 2144 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:06.976894 2068 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:06.981940 3391 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:06.984150 3391 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.986739 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:06.987875 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.989802 2036 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:06.987327 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:06.994100 3234 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:06.995720 3234 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.000516 2648 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:07.009260 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:07.010473 2068 raft_consensus.cc:2396] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:07.034302 2143 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:07.045511 2142 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:07.048935 2070 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
W20260924 00:22:07.049641 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 59869.3us
I20260924 00:22:07.053977 2036 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.059211 2064 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:07.059785 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.061532 3253 consensus_queue.cc:1035] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:07.062656 3376 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:07.063093 3376 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:07.063442 3376 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:07.064605 3421 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.059219 3386 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.072468 2642 raft_consensus.cc:1226] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:07.050038 3311 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:07.073575 3311 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.074335 2139 raft_consensus.cc:2396] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:07.075541 3311 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:06.969687 3348 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.077461 2036 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:07.078059 3421 consensus_queue.cc:1046] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:07.078616 2036 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:07.089069 1992 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:07.078879 3234 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:07.086585 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:07.087364 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:07.088418 2640 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:07.093446 2641 raft_consensus.cc:2379] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:07.090947 3391 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:07.106571 1992 raft_consensus.cc:1261] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:07.101859 3323 consensus_queue.cc:1046] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260924 00:22:07.109114 1992 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:07.110066 3323 raft_consensus.cc:3032] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:07.110430 3323 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:07.111299 3323 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.112514 3323 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.113338 3210 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:07.113762 1987 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:07.085250 3311 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260924 00:22:07.122241 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:07.122898 2143 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:07.123284 2139 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:07.124051 2142 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:07.125061 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.125510 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:07.125910 2140 raft_consensus.cc:2396] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:07.126071 2139 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:07.126415 3210 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:07.126892 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:07.127357 3210 consensus_queue.cc:1046] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:07.107331 3391 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:07.128103 3391 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.128360 3386 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:07.128698 3386 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:07.128804 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:07.129563 3386 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.130704 3386 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.132916 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33238 (request call id 227) took 3274 ms. Trace:
I20260924 00:22:07.133502 3210 consensus_queue.cc:1225] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260924 00:22:07.133817 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.134599 2143 raft_consensus.cc:2379] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:07.134267 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:07.135618 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:07.136582 1992 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:07.135848 3393 raft_consensus.cc:3032] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:07.137456 2144 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:07.138218 2144 raft_consensus.cc:595] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:07.138597 2144 raft_consensus.cc:603] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:07.135237 1892 catalog_manager.cc:5495] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:07.140409 2109 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.140319 3414 consensus_queue.cc:1046] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:07.137815 3393 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:07.141386 3414 consensus_queue.cc:1225] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 9e79063532384ada8ea8f8205478adcf log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260924 00:22:07.141813 3199 raft_consensus.cc:3032] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:07.141902 3414 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:07.142336 3199 raft_consensus.cc:729] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:07.142745 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.143085 3414 raft_consensus.cc:3032] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:07.143603 3414 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:07.143286 3199 consensus_queue.cc:260] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:07.141749 3360 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:07.144677 3199 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:07.144524 3414 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:07.146914 3414 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:07.145527 3393 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:07.149031 3393 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.158730 3376 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.133315 1891 rpcz_store.cc:276] 0924 00:22:03.858401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:03.858898 (+ 497us) service_pool.cc:227] Handling call
0924 00:22:07.132816 (+3273918us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:06.707974 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:06.724553 (+ 16579us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:06.724606 (+ 53us) write_op.cc:435] Acquired schema lock
0924 00:22:06.724625 (+ 19us) tablet.cc:660] Decoding operations
0924 00:22:06.725330 (+ 705us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:06.725423 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:06.725471 (+ 48us) tablet.cc:683] Acquiring locks for 3 operations
0924 00:22:06.725961 (+ 490us) tablet.cc:699] Row locks acquired
0924 00:22:06.726000 (+ 39us) write_op.cc:260] PREPARE: finished
0924 00:22:06.726155 (+ 155us) write_op.cc:270] Start()
0924 00:22:06.726304 (+ 149us) write_op.cc:276] Timestamp: P: 1790209326726096 usec, L: 0
0924 00:22:06.726336 (+ 32us) op_driver.cc:331] REPLICATION: starting
0924 00:22:06.745419 (+ 19083us) log.cc:842] Serialized 1299 byte log entry
0924 00:22:06.944283 (+198864us) op_driver.cc:444] REPLICATION: finished
0924 00:22:07.084365 (+140082us) write_op.cc:301] APPLY: starting
0924 00:22:07.084486 (+ 121us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:07.084655 (+ 169us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:07.084782 (+ 127us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:07.086053 (+ 1271us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:07.086341 (+ 288us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0924 00:22:07.086400 (+ 59us) write_op.cc:312] APPLY: finished
0924 00:22:07.086674 (+ 274us) log.cc:842] Serialized 35 byte log entry
0924 00:22:07.086866 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:07.086947 (+ 81us) write_op.cc:454] Released schema lock
0924 00:22:07.087011 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:07.087273 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:07.087304 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7906,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":132566,"apply.run_cpu_time_us":2664,"apply.run_wall_time_us":4083,"num_ops":3,"prepare.queue_time_us":173179,"prepare.run_cpu_time_us":4019,"prepare.run_wall_time_us":76084,"replication_time_us":216063,"thread_start_us":198449,"threads_started":2}]]}
W20260924 00:22:07.164776 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 105093us
I20260924 00:22:07.160892 3376 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:07.172227 3348 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:07.172590 3429 ts_tablet_manager.cc:927] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:07.166222 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.175575 3380 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.176812 2109 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.168444 3234 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:07.169824 3391 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.178962 3234 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.179842 3391 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:07.182453 2064 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:22:07.183341 3376 consensus_queue.cc:1046] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:07.184649 3040 raft_consensus.cc:3032] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:07.184739 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:07.185015 3040 raft_consensus.cc:729] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:07.185885 3040 consensus_queue.cc:260] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:07.180032 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 126674us
I20260924 00:22:07.180954 2109 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.181895 3424 consensus_queue.cc:1046] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:07.186817 3040 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:07.186733 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:07.187824 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:07.188731 2065 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:07.193564 3199 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:07.206272 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.207134 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:07.206566 2068 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: [1.2-3.3]
I20260924 00:22:07.204101 1892 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:07.208387 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:07.221341 2137 raft_consensus.cc:2396] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:07.215545 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.222259 2143 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.218313 2066 raft_consensus.cc:1226] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:07.218149 3424 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:07.219178 2071 raft_consensus.cc:1226] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:07.225826 2067 raft_consensus.cc:1261] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:07.225924 3424 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:07.229694 2109 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.231235 2109 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.227139 3424 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.236433 3353 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:07.238987 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.198787 3323 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.233493 2143 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:07.227828 3376 raft_consensus.cc:3032] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:07.248875 3376 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:07.250108 3380 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:07.249883 3376 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:07.251957 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:07.252621 2143 raft_consensus.cc:595] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.253155 2143 raft_consensus.cc:603] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:07.247402 1892 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:07.240892 3199 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:07.266961 2035 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.246487 3338 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:07.245769 3393 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:07.266012 1892 catalog_manager.cc:5495] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:07.273453 2137 raft_consensus.cc:1261] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:22:07.273914 1892 catalog_manager.cc:5495] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:07.276032 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91be317234ca451fad76824aef0861a6"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:07.276672 2143 raft_consensus.cc:595] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.212649 3429 tablet_copy_client.cc:316] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:22:07.276999 2143 raft_consensus.cc:603] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:07.274989 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.278764 3423 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:07.250939 3392 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260924 00:22:07.246907 3386 consensus_queue.cc:1046] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260924 00:22:07.279433 3300 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:07.279919 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.275336 1892 catalog_manager.cc:5495] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 2 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:07.283533 3312 consensus_queue.cc:1035] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:07.284682 2067 raft_consensus.cc:3032] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:07.285158 2067 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:07.284757 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:22:07.285890 2067 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.286876 2067 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.270259 3338 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:07.290290 1892 catalog_manager.cc:5495] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 3 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:07.280705 3376 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:07.295604 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:07.296119 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:07.293242 3300 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:07.298184 3253 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:07.298271 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:07.296903 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:07.295257 2035 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.292613 1892 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:07.306480 3040 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:07.307476 3040 raft_consensus.cc:686] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:07.309161 3040 consensus_queue.cc:237] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.310379 3391 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.293735 3338 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:07.294275 3424 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:07.338456 1992 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260924 00:22:07.338721 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 43550.2us
I20260924 00:22:07.339480 2035 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.340534 2035 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.298226 3300 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.355422 3391 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:07.356132 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet d2ef9fa9e5dd48218a5c243de0c0e49a from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:22:07.356755 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet d2ef9fa9e5dd48218a5c243de0c0e49a from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-d2ef9fa9e5dd48218a5c243de0c0e49a
I20260924 00:22:07.362164 3338 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.364183 3338 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:07.305413 1892 catalog_manager.cc:5495] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } }
I20260924 00:22:07.365875 1993 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:22:07.366098 2071 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:07.366477 1892 catalog_manager.cc:5495] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 2 to 3, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 3 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:07.368278 3210 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260924 00:22:07.370689 3430 log.cc:924] Time spent T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260924 00:22:07.370988 2640 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:22:07.373585 2035 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.374369 3427 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:07.376662 3393 raft_consensus.cc:3032] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:07.376775 2035 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:07.377179 3393 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:07.377640 3427 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:07.378065 3427 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:07.378489 3427 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.378100 3393 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.379451 3393 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.382773 3348 consensus_queue.cc:1046] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:07.384286 3423 raft_consensus.cc:3032] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:07.384626 3423 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:07.384410 3392 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:07.385587 3210 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:07.385522 3423 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.386760 3423 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.387961 3323 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:07.368340 1892 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:07.395752 2067 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260924 00:22:07.396275 2067 pending_rounds.cc:76] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 1
I20260924 00:22:07.396581 2067 pending_rounds.cc:97] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 3.2
W20260924 00:22:07.404830 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 106619us
I20260924 00:22:07.406750 2155 tablet_copy_source_session.cc:205] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:07.409018 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:07.418709 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.422197 2067 raft_consensus.cc:2866] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:07.423074 3040 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:07.423715 3040 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:07.424360 3040 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.431409 3040 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.433650 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:07.434933 3348 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:07.417407 3391 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:07.437294 3392 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:07.463376 2035 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.455025 3253 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:07.446285 3427 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.470382 2035 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.471369 2144 raft_consensus.cc:1226] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:07.465616 3376 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:07.481578 3040 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:07.484972 1995 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260924 00:22:07.486941 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.468766 3348 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.485520 1995 pending_rounds.cc:76] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 1
I20260924 00:22:07.515720 1995 pending_rounds.cc:97] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260924 00:22:07.516146 1995 raft_consensus.cc:2866] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:07.517561 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.528975 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.534680 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:07.535956 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 49098.3us
I20260924 00:22:07.537539 2035 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:07.539647 2142 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:07.541716 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:07.546999 2065 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:07.513257 3234 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.513011 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.516938 3300 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:07.551287 1990 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:07.551740 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.514904 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:07.517530 3426 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.553661 3311 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:07.557332 1995 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:07.515266 3376 raft_consensus.cc:686] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:07.553033 3253 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.566084 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:07.569653 1994 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.574280 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:07.556471 3424 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:07.535183 3210 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260924 00:22:07.585227 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.592474 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.593997 3427 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:07.595243 2145 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260924 00:22:07.595621 2145 pending_rounds.cc:76] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Aborting all ops after (but not including) 1
I20260924 00:22:07.595819 2145 pending_rounds.cc:97] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.2
I20260924 00:22:07.596069 2145 raft_consensus.cc:2894] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Aborting config change with OpId 1.2: Aborted: Op aborted by new leader
I20260924 00:22:07.596429 2145 raft_consensus.cc:2866] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:07.597043 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:39160 (request call id 4) took 9751 ms. Trace:
I20260924 00:22:07.597381 3380 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:07.597832 3380 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:07.598263 3380 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.599563 3376 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.608316 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:07.581795 3426 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:07.612169 1994 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:07.584396 3311 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:07.612100 3441 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:07.618829 3439 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:07.591626 3424 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:07.608378 3427 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:07.626950 2144 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:07.628170 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:07.618817 1989 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:07.629289 3426 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.568156 3300 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:07.634217 3443 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20260924 00:22:07.665211 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 107185us
I20260924 00:22:07.666373 2067 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:07.669723 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:07.669579 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 96286.2us
I20260924 00:22:07.683314 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:07.597383 2145 rpcz_store.cc:276] 0924 00:21:57.848906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:21:57.849015 (+ 109us) service_pool.cc:227] Handling call
0924 00:21:57.854466 (+ 5451us) log.cc:842] Serialized 492 byte log entry
0924 00:21:58.009384 (+154918us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b18000f2740. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d1b0b4ca 00007fb6db1ecf99 00007fb6d19cf990 00007fb6d1b91864 00007fb6d1b91155 00007fb6d4a36320 00007fb6d4a9c113 00007fb6d4ac27e2 00007fb6d99fef39
0924 00:21:58.028860 (+ 19476us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b2c000c2d48. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d349fe11 00007fb6d34aeada 00007fb6d34ae763 00007fb6d34ae701 00007fb6d4af8d62 00007fb6d4af8f05 00007fb6d4a4df91 00007fb6d4a36545 00007fb6d4a3b771 00007fb6d4a3b6f9 00007fb6d4a3b689
0924 00:21:58.030289 (+ 1429us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b1800023fa0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4af8f11 00007fb6d4a4df91 00007fb6d4a36545 00007fb6d4a3b771 00007fb6d4a3b6f9 00007fb6d4a3b689 00007fb6d4a3b651 00007fb6d4a3a81d 00007fb6dadf8204 00007fb6dadf8039
0924 00:22:07.596953 (+9566664us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90449,"raft.queue_time_us":88174,"raft.run_cpu_time_us":3835,"raft.run_wall_time_us":91258,"spinlock_wait_cycles":23508736,"thread_start_us":75501,"threads_started":1,"wal-append.queue_time_us":77288,"wal-append.run_cpu_time_us":1686,"wal-append.run_wall_time_us":1012984}
I20260924 00:22:07.685681 2145 pending_rounds.cc:97] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260924 00:22:07.685967 2145 raft_consensus.cc:2866] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260924 00:22:07.687451 1942 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted by new leader
I20260924 00:22:07.690101 2137 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:07.664116 3427 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.700397 2141 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260924 00:22:07.700928 2141 pending_rounds.cc:76] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Aborting all ops after (but not including) 1
I20260924 00:22:07.701216 2141 pending_rounds.cc:97] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260924 00:22:07.682499 3353 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:07.701516 2141 raft_consensus.cc:2866] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260924 00:22:07.765051 3446 log.cc:924] Time spent T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
I20260924 00:22:07.767392 3429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ef9fa9e5dd48218a5c243de0c0e49a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 00:22:07.773693 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 61215.3us
W20260924 00:22:07.913410 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 143834us
W20260924 00:22:07.923959 3457 log.cc:924] Time spent T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.213s user 0.000s sys 0.000s
I20260924 00:22:07.894507 3441 raft_consensus.cc:686] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
W20260924 00:22:07.927094 3390 log.cc:924] Time spent T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Append to log took a long time: real 0.261s user 0.000s sys 0.001s
I20260924 00:22:07.711457 3353 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:07.709268 3311 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:07.954072 3234 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:07.926499 3421 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:07.974570 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 329457us
I20260924 00:22:07.981945 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:07.983814 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.988686 3040 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:07.990362 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:07.974877 3300 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:08.001163 3447 log.cc:924] Time spent T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.226s user 0.021s sys 0.007s
I20260924 00:22:08.001430 3426 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.002655 3300 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:08.003145 3300 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:08.004371 3300 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.004763 3234 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.007002 3234 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:08.007529 3463 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:08.011154 3380 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.013558 3441 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.018275 1993 raft_consensus.cc:1226] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:07.970620 3452 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:08.033955 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 132292us
I20260924 00:22:08.040253 3338 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:08.003351 3461 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.346s
I20260924 00:22:08.052953 3253 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.055994 3380 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.041416 3452 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.034237 3421 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.111414 1891 catalog_manager.cc:5495] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:08.085672 3427 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:08.121802 3467 log.cc:924] Time spent T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260924 00:22:08.111832 3465 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.094583 3353 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.094959 3414 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.092145 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:08.106710 3452 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.119611 3360 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:08.125469 3391 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:08.151687 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.152294 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.152431 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:08.153594 3456 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.218s user 0.000s sys 0.003s
I20260924 00:22:08.177630 2142 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:08.146325 3210 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:08.180302 2145 raft_consensus.cc:2379] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:08.180696 3391 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.182866 3391 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.183825 3360 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:08.186069 3424 consensus_meta.cc:287] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.524s user 0.009s sys 0.001s
I20260924 00:22:08.186450 3424 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:08.090108 3338 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.092564 3253 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.177961 3040 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
W20260924 00:22:08.178301 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 176142us
I20260924 00:22:08.155167 3332 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.176981 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:08.197959 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d2ef9fa9e5dd48218a5c243de0c0e49a, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.197959 2064 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
W20260924 00:22:08.198527 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d2ef9fa9e5dd48218a5c243de0c0e49a, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.200065 3421 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:08.178185 3323 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.201465 3323 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.202493 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.202906 3253 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.203280 2137 raft_consensus.cc:2379] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:08.170463 3465 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.204077 2140 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:08.204023 3470 consensus_queue.cc:1046] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:08.205080 3199 raft_consensus.cc:3032] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:08.205446 3199 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:08.205518 3465 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.206171 3465 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.206303 3199 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:08.207476 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 24240f07434f49e9bef897dac3fbe801, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:08.208779 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 24240f07434f49e9bef897dac3fbe801, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.208820 3338 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.209002 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.209753 1991 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:08.203536 3323 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.171041 3414 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.207571 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.206629 3465 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.215325 2035 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:08.218894 1994 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:08.186215 3360 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:08.226629 2110 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:08.228438 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.229831 3040 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.232278 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.233013 2064 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:08.233013 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.233762 2070 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:08.244776 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.245685 1990 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:08.248688 2614 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:08.250463 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 100830us
I20260924 00:22:08.254648 2110 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:08.270212 3393 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:08.272045 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.241747 3427 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:08.277510 3353 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:08.258737 3360 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:08.289786 3360 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:08.290258 3360 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:08.212128 3332 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.298884 3332 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.299556 2640 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:08.212800 3210 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.301985 3210 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.280719 3465 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.304204 3427 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.304690 3427 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:08.309449 3353 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.310009 3353 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.314128 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:08.314989 2641 raft_consensus.cc:2379] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:08.317476 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.218676 3414 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.284071 2109 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:08.319653 3479 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:08.320084 3479 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.320457 3479 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:08.284863 3393 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.321357 3393 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.268646 3481 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:08.289218 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:08.317588 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.320210 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.287940 1990 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:08.326296 2035 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:08.338608 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.339212 2068 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:08.339985 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.326622 2141 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:08.288552 3199 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.330207 2649 raft_consensus.cc:2379] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:08.342118 2613 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:08.336969 2064 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:08.333683 2036 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:08.346171 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.346388 2067 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260924 00:22:08.347179 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:08.347347 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.347761 2144 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.344148 2062 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.347930 1994 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:08.349611 3423 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:08.348501 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.346825 1993 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:08.353299 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:08.356829 2069 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:08.349896 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:08.358084 2063 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.359412 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:08.361205 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:08.340255 3360 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.367508 3478 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:08.368065 3478 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.368876 3360 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:08.369431 3312 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:08.369808 3312 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.370106 3312 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.370229 3478 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.371563 3426 consensus_queue.cc:1046] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:08.371927 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.372575 3426 raft_consensus.cc:3032] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:08.372910 3426 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:08.372025 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.373592 2138 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:08.377319 3312 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.377943 2613 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:08.370131 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:08.378957 2649 raft_consensus.cc:2379] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:08.378899 3462 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:08.379475 3462 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.379873 3462 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.380530 2144 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:08.380220 3427 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.382019 3479 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.383509 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:08.383945 3479 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.384202 2143 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:08.344851 3210 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.385932 3210 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.387005 3462 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.387951 3210 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.379451 3312 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.373654 3426 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.389940 3462 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.390643 3426 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:08.392154 3427 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.391465 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:08.392966 2068 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.363328 3234 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:08.395920 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:08.396504 2144 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.391567 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:08.398802 3393 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.406088 2614 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:08.406684 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.407049 3462 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:08.407318 1993 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.407521 3462 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.407963 3462 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.364449 3391 consensus_queue.cc:1046] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260924 00:22:08.409258 3391 raft_consensus.cc:3032] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:08.411958 2136 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:08.413547 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.414227 1988 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.416543 3462 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.420440 2068 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:08.394801 2143 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:08.424480 2035 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:08.426553 1993 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:08.422241 2144 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:08.428469 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.330722 3441 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260924 00:22:08.429687 3462 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.429045 1993 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.430665 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.430894 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.432060 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.362167 3348 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:08.356272 3423 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.425061 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:08.422780 2613 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:08.409634 3391 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:08.433077 2110 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:08.434005 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.439086 2062 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:08.474983 2110 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:08.455191 2613 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:08.455480 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:08.461326 2140 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:08.423367 2063 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:08.433467 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.457036 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.463296 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.486064 2141 raft_consensus.cc:2354] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:08.433709 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.463485 2035 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:08.453706 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.434321 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.463665 3348 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:08.473335 2067 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:08.452239 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:08.485837 1994 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.440737 1993 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:08.439344 3423 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.499096 3040 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260924 00:22:08.499246 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 47093.6us
I20260924 00:22:08.502708 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:08.455546 3476 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:08.504145 3476 raft_consensus.cc:686] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:08.502583 3481 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:08.504864 3476 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.506474 3424 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:08.507086 3424 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:08.507508 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:08.508195 3300 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:08.508680 3300 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:08.462201 3479 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.509290 3486 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.453020 3482 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.478546 3478 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:08.478721 3210 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:08.499650 2110 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:08.434783 3393 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.475833 3234 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.490437 3462 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:08.497519 3461 raft_consensus.cc:2741] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260924 00:22:08.527266 2139 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.521114 2067 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:08.451894 3253 consensus_queue.cc:1046] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260924 00:22:08.521678 3040 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:08.535338 1994 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:08.537231 2110 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:08.502080 2064 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:08.516605 3348 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:08.509433 2614 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:08.530598 3360 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:08.547348 2109 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:08.549290 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:08.550974 2139 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:08.509578 3479 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.554837 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:08.462788 3391 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.539769 3462 raft_consensus.cc:686] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:08.545351 3478 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:08.552398 3353 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.676776 3040 raft_consensus.cc:3032] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:08.562836 3441 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.119s
I20260924 00:22:08.682066 3040 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:08.683200 3348 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.646328 3253 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:08.684494 3253 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.686543 3253 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.683355 3040 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.687395 3040 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:08.687430 3462 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.517619 3486 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.690256 3424 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260924 00:22:08.643559 3456 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260924 00:22:08.691635 3477 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:08.692216 3477 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.649576 3210 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:08.645957 1960 leader_election.cc:398] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
I20260924 00:22:08.693441 1960 leader_election.cc:401] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:08.694384 3477 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.681541 3391 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:08.509521 3300 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.698992 3353 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.700542 3486 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.701478 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:08.702294 2064 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.707855 3414 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.708320 3479 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.708411 3414 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.710638 3414 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:08.710211 3423 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.711318 3429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45582 (request call id 158) took 1624 ms. Trace:
I20260924 00:22:08.711753 2068 raft_consensus.cc:2379] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:08.712620 3426 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.713441 3423 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.714787 1988 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:08.715433 3478 raft_consensus.cc:686] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:08.715507 3426 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.520691 3482 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.716490 3478 consensus_queue.cc:237] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.533365 3234 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.721244 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.721926 2067 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:08.724253 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.724596 3482 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:08.724900 2069 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:22:08.731555 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 171210us
I20260924 00:22:08.732278 3439 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3.
W20260924 00:22:08.734357 3491 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.196s user 0.000s sys 0.000s
I20260924 00:22:08.734756 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.736007 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.736860 2071 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:08.740048 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:08.741096 2109 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:08.722609 2142 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:08.560364 3376 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:08.744009 3479 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:08.745107 1989 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260924 00:22:08.746660 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.747150 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.747838 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.749624 1993 raft_consensus.cc:2379] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:08.646272 3312 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:08.646533 3360 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260924 00:22:08.560242 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 50911.1us
I20260924 00:22:08.679476 3483 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:08.755733 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.756251 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.759884 1991 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:08.760210 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:08.742899 3210 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:08.750531 3479 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.769011 2067 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.770967 2066 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.771705 2036 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:08.778255 1993 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:08.711719 3429 rpcz_store.cc:276] 0924 00:22:07.087245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:07.090868 (+ 3623us) service_pool.cc:227] Handling call
0924 00:22:07.173017 (+ 82149us) ts_tablet_manager.cc:928] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
0924 00:22:07.690667 (+517650us) tablet_copy_client.cc:361] Tablet copy session begun
0924 00:22:08.296560 (+605893us) tablet_metadata.cc:641] Metadata flushed
0924 00:22:08.296828 (+ 268us) tablet_copy_client.cc:490] Wrote new tablet metadata
0924 00:22:08.710468 (+413640us) tablet_copy_client.cc:411] Wrote new consensus metadata
0924 00:22:08.710794 (+ 326us) tablet_replica.cc:205] Creating consensus instance
0924 00:22:08.711154 (+ 360us) ts_tablet_manager.cc:983] Replica registered.
0924 00:22:08.711221 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":340,"tablet-copy.queue_time_us":78087,"thread_start_us":108800,"threads_started":2}
I20260924 00:22:08.780972 2649 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260924 00:22:08.781252 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.753609 3199 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.783516 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.784216 1994 raft_consensus.cc:2379] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:08.786661 2070 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260924 00:22:08.786916 3040 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:08.761835 3392 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.756171 3470 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260924 00:22:08.791904 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 54b1b238378448a5987c49aafc3b6313, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:08.794507 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 54b1b238378448a5987c49aafc3b6313, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.757020 3360 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.765690 3312 raft_consensus.cc:686] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:08.761008 3376 raft_consensus.cc:686] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
W20260924 00:22:08.647492 3427 consensus_queue.cc:1159] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8c129c01e1c047628f6a87b001aa7258" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260924 00:22:08.762379 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.792165 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:08.766685 3414 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.763919 3423 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:08.802326 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:08.816869 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.798830 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:08.796016 3040 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260924 00:22:08.776660 3468 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.782956 3479 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:08.810176 2036 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:08.793440 3311 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:08.793552 3473 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:08.791105 3386 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.819377 2069 raft_consensus.cc:2379] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:08.818787 2067 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:08.794431 3470 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.807647 3360 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:08.847330 3040 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260924 00:22:08.803249 3429 tablet_copy_client.cc:791] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:08.819183 1987 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:08.851446 2063 raft_consensus.cc:1226] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:08.792928 3199 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.826371 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:08.855967 3464 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:08.857023 3271 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:08.861696 3475 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:08.863198 3429 tablet_copy_client.cc:655] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:08.848681 3210 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.841907 3421 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:08.841390 3386 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.835616 3473 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:08.828523 3423 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.822894 3414 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.832868 3468 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.765954 3483 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.846156 3470 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.875012 3312 consensus_queue.cc:237] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.876016 3376 consensus_queue.cc:237] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.881438 3426 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.882429 3234 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.884701 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:08.933207 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 48580us
I20260924 00:22:08.798444 3392 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:08.830485 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:08.899667 2066 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260924 00:22:08.910699 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:08.940155 1993 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.924708 3468 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:08.911634 3473 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:08.959586 2142 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:08.888705 3479 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.976892 1987 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:08.899518 3271 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:08.996459 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 186744us
I20260924 00:22:08.901425 3475 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:08.939402 3441 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:08.951123 3477 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:08.990167 2139 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:08.998970 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:08.964178 3483 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.996801 1993 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:08.960291 2136 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:09.005543 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:09.010910 3040 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:09.015638 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.016669 1993 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:09.020655 2067 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260924 00:22:09.028409 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:09.033818 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.041997 2069 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:09.032847 2649 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:09.005585 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:09.054723 3502 log.cc:924] Time spent T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260924 00:22:09.054807 2064 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
W20260924 00:22:09.056244 3503 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260924 00:22:09.057976 3423 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:08.999329 3468 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260924 00:22:09.029395 3348 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:09.061794 3468 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:09.062211 3468 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:09.013805 3338 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260924 00:22:09.063091 3468 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.051401 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.023824 3478 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:09.031242 3476 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:09.070873 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.074298 3464 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:09.031905 2640 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:09.068297 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.081729 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:09.066071 3501 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260924 00:22:09.065107 3468 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:09.063071 3271 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:09.096271 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.065478 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.084110 3426 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20260924 00:22:09.098694 3480 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:09.109364 3480 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:09.099812 3497 log.cc:924] Time spent T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260924 00:22:09.100311 3392 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:09.066563 3486 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:09.103186 3482 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:09.076673 3199 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.105711 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.094695 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.088027 3464 raft_consensus.cc:686] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:09.069725 2614 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:09.095533 3386 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.079430 3461 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260924 00:22:09.075906 3478 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.124672 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.135531 2142 raft_consensus.cc:2379] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:09.117884 3414 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:09.078493 3476 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:09.125861 2137 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:09.131570 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.121520 2136 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:09.102216 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:09.118036 3482 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:09.155716 3482 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:09.126910 3311 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:09.157433 3311 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.157524 3464 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.121251 3486 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.128314 3479 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:09.140982 3504 log.cc:924] Time spent T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260924 00:22:09.141408 3475 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.141737 3353 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:09.106192 3253 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260924 00:22:09.144145 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.151611 1990 raft_consensus.cc:2354] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:09.152544 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:09.133426 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 48572.8us
I20260924 00:22:09.140960 3480 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:09.160553 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 58437.1us
W20260924 00:22:09.167685 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 99246.7us
I20260924 00:22:09.168650 2142 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:09.163867 3380 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:09.203251 3380 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:09.203682 3380 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:09.203912 3473 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.205063 2140 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:09.170399 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:09.209553 2142 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:09.211853 3421 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:09.216313 3311 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:09.187505 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.218647 3473 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.190479 2062 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260924 00:22:09.170614 3353 raft_consensus.cc:686] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:09.220968 2064 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:09.191092 2137 raft_consensus.cc:2379] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:09.193223 3478 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.195565 3360 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.197252 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:09.163009 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 117065us
I20260924 00:22:09.235700 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.169777 3476 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.202776 3486 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.227576 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.230997 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.231819 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.238142 2143 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260924 00:22:09.243180 2143 pending_rounds.cc:76] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Aborting all ops after (but not including) 1
I20260924 00:22:09.243428 2143 pending_rounds.cc:97] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260924 00:22:09.243666 2143 raft_consensus.cc:2866] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:09.248845 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.248697 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.252523 2145 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
W20260924 00:22:09.252305 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 160764us
I20260924 00:22:09.233307 3452 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:09.258213 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:09.261611 2109 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.263980 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.265867 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.267145 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.267719 2067 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:09.267665 2035 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:09.267954 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.269488 2035 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:09.238323 3421 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:09.220607 3482 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.275490 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.317528 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.306311 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.305776 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.320734 2065 raft_consensus.cc:2379] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:09.251754 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:09.321689 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f2f86eadee0b4bbcbfd3437d30d304b5, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.323374 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:09.323794 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f2f86eadee0b4bbcbfd3437d30d304b5, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:09.326025 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 65091.5us
I20260924 00:22:09.265563 3452 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.305301 3311 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:09.276662 3426 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:09.379069 2109 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:09.264760 3475 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:22:09.376288 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 49488.9us
I20260924 00:22:09.323060 3429 tablet_copy_client.cc:531] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:09.381717 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:09.385710 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 52761us
I20260924 00:22:09.331137 3520 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:09.409804 3473 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:09.410378 3473 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:09.410499 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:09.410868 3473 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:09.275991 3441 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.107s
I20260924 00:22:09.412017 2109 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.413662 2109 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.415159 3521 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:09.416750 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:09.417798 2649 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:09.417893 2109 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.420048 3521 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:09.420684 3482 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:09.421815 3452 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:09.465175 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 50243us
W20260924 00:22:09.465592 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 75555.5us
W20260924 00:22:09.467952 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 50164.7us
I20260924 00:22:09.468688 2109 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.379173 3311 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:09.323276 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:09.493307 3523 log.cc:924] Time spent T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260924 00:22:09.499024 3468 consensus_queue.cc:1046] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:09.464665 3414 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:09.502177 3253 raft_consensus.cc:3032] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:09.502555 3253 raft_consensus.cc:729] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:09.503291 3253 consensus_queue.cc:260] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.332015 3514 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.153s
W20260924 00:22:09.467813 3491 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.092s user 0.000s sys 0.009s
I20260924 00:22:09.374565 3470 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:09.512915 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 44294.3us
I20260924 00:22:09.326465 3421 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.518244 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.520504 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.527666 2141 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:09.467195 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.330574 3353 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:09.473264 3311 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:09.102713 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:09.499444 3473 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.596858 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:09.506021 3253 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:09.516749 3521 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:09.383735 3498 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.129s
I20260924 00:22:09.601675 2063 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:09.519858 3392 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:09.464882 2640 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:09.519155 3360 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:09.512149 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 213743us
I20260924 00:22:09.528391 3479 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:09.466177 3520 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.645509 2063 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:09.600077 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:09.519629 3380 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:09.657807 2109 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:09.606508 3477 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:09.641755 3312 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:09.609459 3475 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:09.665365 2109 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:09.667804 1993 raft_consensus.cc:2379] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:09.610733 3521 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.670261 3471 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:09.670689 3471 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:09.671031 3471 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:09.674568 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 351169us
I20260924 00:22:09.679780 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.680591 1994 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:09.683445 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:09.689796 3525 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260924 00:22:09.691342 3470 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:09.694620 3522 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.221s user 0.002s sys 0.000s
I20260924 00:22:09.701324 1995 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:09.702184 3521 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.703168 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.703773 1992 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:09.704190 3486 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:09.705343 2110 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:09.706127 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.706561 3478 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:09.706287 1988 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:09.706774 1990 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:09.706950 3478 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:09.707346 3478 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:09.652657 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.709478 3312 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:09.709931 3312 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:09.657254 3479 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:09.710506 3479 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.609015 3481 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260924 00:22:09.714380 3414 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:09.666175 3505 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260924 00:22:09.715828 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:09.715760 3478 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.717758 3520 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.717864 3478 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.720361 3380 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:09.722256 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 126) took 2048 ms. Trace:
W20260924 00:22:09.731388 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 136695us
I20260924 00:22:09.712898 3479 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.742260 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:09.754657 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:09.755519 2063 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:09.755700 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.756294 2069 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:09.722612 1891 rpcz_store.cc:276] 0924 00:22:07.691599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:07.695807 (+ 4208us) service_pool.cc:227] Handling call
0924 00:22:09.722173 (+2026366us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:08.181115 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:08.181600 (+ 485us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:08.181632 (+ 32us) write_op.cc:435] Acquired schema lock
0924 00:22:08.181644 (+ 12us) tablet.cc:660] Decoding operations
0924 00:22:08.181954 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:08.181998 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:08.182026 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:08.182193 (+ 167us) tablet.cc:699] Row locks acquired
0924 00:22:08.182211 (+ 18us) write_op.cc:260] PREPARE: finished
0924 00:22:08.291097 (+108886us) spinlock_profiling.cc:243] Waited 7.24 ms on lock 0x7b2c00126798. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d349fe11 00007fb6d34aeada 00007fb6d34ae763 00007fb6d34ae701 00007fb6d4af7b12 00007fb6d4af78ed 00007fb6d84cb7a0 00007fb6d84cb2b8 00007fb6d84cfba1 00007fb6d84cfb59 00007fb6d84cfae9
0924 00:22:08.291167 (+ 70us) write_op.cc:270] Start()
0924 00:22:08.291411 (+ 244us) write_op.cc:276] Timestamp: P: 1790209328189539 usec, L: 0
0924 00:22:08.291439 (+ 28us) op_driver.cc:331] REPLICATION: starting
0924 00:22:08.292063 (+ 624us) log.cc:842] Serialized 530 byte log entry
0924 00:22:08.447407 (+155344us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b54002a0920. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41af0 00007fb6d4a4c028 00007fb6d4a4bc6a 00007fb6d4ab7cd5 00007fb6d4ab8179 00007fb6d84cbade 00007fb6d84cb2b8 00007fb6d84cfba1 00007fb6d84cfb59 00007fb6d84cfae9 00007fb6d84cfab1
0924 00:22:08.927365 (+479958us) op_driver.cc:444] REPLICATION: finished
0924 00:22:08.928199 (+ 834us) write_op.cc:301] APPLY: starting
0924 00:22:08.928309 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:08.928459 (+ 150us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:08.928493 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:08.928778 (+ 285us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:08.929000 (+ 222us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:08.929072 (+ 72us) write_op.cc:312] APPLY: finished
0924 00:22:08.929378 (+ 306us) log.cc:842] Serialized 23 byte log entry
0924 00:22:08.929583 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:08.929686 (+ 103us) write_op.cc:454] Released schema lock
0924 00:22:08.929736 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:08.930052 (+ 316us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:08.930092 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":598083,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":537,"apply.run_cpu_time_us":2147,"apply.run_wall_time_us":2147,"num_ops":1,"prepare.queue_time_us":3809,"prepare.run_cpu_time_us":7698,"prepare.run_wall_time_us":270444,"replication_time_us":635799,"spinlock_wait_cycles":113946880}]]}
I20260924 00:22:09.762557 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33232 (request call id 215) took 2769 ms. Trace:
I20260924 00:22:09.772220 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.772267 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:09.772933 2066 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:09.776160 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:09.775290 2063 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:09.794942 1992 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:09.795753 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:09.797570 2109 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:09.802484 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:09.804014 2109 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:09.772213 3475 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:09.763577 1892 rpcz_store.cc:276] 0924 00:22:06.993701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:06.993845 (+ 144us) service_pool.cc:227] Handling call
0924 00:22:07.055438 (+ 61593us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x7b5400002dd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41af0 00007fb6d4ac4f2a 00007fb6daa16d6c 00007fb6dab10c3e 00007fb6d8e22fb4 00007fb6d8e22f42 00007fb6d8e22e71 00007fb6d8e22dec 00007fb6d8e220a2 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:09.762490 (+2707052us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:07.617921 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:07.618384 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:07.618417 (+ 33us) write_op.cc:435] Acquired schema lock
0924 00:22:07.618434 (+ 17us) tablet.cc:660] Decoding operations
0924 00:22:07.625367 (+ 6933us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:07.625447 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:07.625474 (+ 27us) tablet.cc:683] Acquiring locks for 11 operations
0924 00:22:07.626488 (+ 1014us) tablet.cc:699] Row locks acquired
0924 00:22:07.626509 (+ 21us) write_op.cc:260] PREPARE: finished
0924 00:22:07.626670 (+ 161us) write_op.cc:270] Start()
0924 00:22:07.626783 (+ 113us) write_op.cc:276] Timestamp: P: 1790209327626619 usec, L: 0
0924 00:22:07.626802 (+ 19us) op_driver.cc:331] REPLICATION: starting
0924 00:22:07.643606 (+ 16804us) log.cc:842] Serialized 4352 byte log entry
0924 00:22:08.448248 (+804642us) op_driver.cc:444] REPLICATION: finished
0924 00:22:08.877765 (+429517us) write_op.cc:301] APPLY: starting
0924 00:22:08.883276 (+ 5511us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:08.883561 (+ 285us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:08.888548 (+ 4987us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:08.893672 (+ 5124us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:08.893976 (+ 304us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0924 00:22:08.894045 (+ 69us) write_op.cc:312] APPLY: finished
0924 00:22:08.894425 (+ 380us) log.cc:842] Serialized 83 byte log entry
0924 00:22:08.895590 (+ 1165us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:08.895773 (+ 183us) write_op.cc:454] Released schema lock
0924 00:22:08.895836 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:08.898322 (+ 2486us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:08.898369 (+ 47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4556,"reported_tablets":25,"spinlock_wait_cycles":117411072,"child_traces":[["op",{"apply.queue_time_us":392479,"apply.run_cpu_time_us":4641,"apply.run_wall_time_us":47839,"num_ops":11,"prepare.queue_time_us":108627,"prepare.run_cpu_time_us":4531,"prepare.run_wall_time_us":45036,"replication_time_us":821344,"spinlock_wait_cycles":3456,"thread_start_us":358940,"threads_started":2}]]}
I20260924 00:22:09.668696 3477 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.810662 3529 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:09.811192 3529 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:09.811625 3529 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:09.810406 3393 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:09.818387 3393 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:09.819199 3393 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.787384 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.821852 1989 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:09.822290 3429 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:09.811369 2064 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:09.824213 3473 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:09.811807 2138 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260924 00:22:09.824074 3529 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:09.825215 3421 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:09.763972 3311 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.827628 3471 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:09.828578 3311 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:09.785974 3452 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:09.829406 3452 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:09.830431 3529 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:09.831640 3452 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:09.832000 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:09.839845 1989 raft_consensus.cc:2379] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:09.839695 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.840329 2145 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:09.840664 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:09.841288 2141 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:09.841429 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:09.842079 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:09.842756 2640 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:09.831662 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:09.844515 1992 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:09.832693 2614 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:09.793277 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:09.835090 1995 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:09.836351 3471 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:09.846521 2035 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:09.848790 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:09.808812 3414 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:09.808429 3521 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260924 00:22:09.853802 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 329135us
I20260924 00:22:09.855103 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:09.862987 2064 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:09.847997 3452 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:09.875216 1992 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:09.878054 2110 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:09.861434 3311 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:09.884620 2145 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:09.884951 3311 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:09.885409 3311 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:09.885876 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 53274.9us
I20260924 00:22:09.886786 2614 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.867179 3454 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260924 00:22:09.893757 2036 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:09.895030 2036 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:09.889645 3464 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:09.896947 2062 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:09.899320 2065 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:09.902630 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:09.907315 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:09.908056 2142 raft_consensus.cc:2379] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:09.908771 2036 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:09.910246 2036 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:09.912647 2064 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:09.901444 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:09.852511 3475 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:09.856316 3414 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:09.917922 2063 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:09.925215 2141 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:09.918306 2069 raft_consensus.cc:2379] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:09.918883 2064 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:09.923386 2070 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:09.865502 3521 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:09.939810 2140 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:09.939857 2145 raft_consensus.cc:1261] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:09.934866 2067 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260924 00:22:09.935339 2139 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:09.936539 2065 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:09.936940 1960 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:09.942147 2070 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260924 00:22:09.930372 2143 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:09.934062 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:09.907495 3391 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:09.901891 3421 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:22:09.946540 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 45640.7us
I20260924 00:22:09.913869 3479 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:09.955749 2064 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:09.935154 3414 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:09.938922 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:09.944613 3507 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:09.944715 3521 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:09.945230 3481 raft_consensus.cc:2741] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260924 00:22:09.964668 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 50543.6us
I20260924 00:22:09.970458 2035 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:09.946772 3480 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:09.913595 3452 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.062669 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1062) took 1272 ms. Trace:
I20260924 00:22:09.946398 3500 consensus_queue.cc:1046] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260924 00:22:09.965494 3479 raft_consensus.cc:686] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:10.021095 3475 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:10.021869 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 44850us
I20260924 00:22:09.982482 1988 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:10.027999 3312 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:09.956984 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 70297.1us
I20260924 00:22:10.033267 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:09.975862 3353 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260924 00:22:10.061357 2109 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:10.082851 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 112518us
I20260924 00:22:10.086218 3480 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.101660 1989 raft_consensus.cc:595] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:09.978361 3376 consensus_queue.cc:1046] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20260924 00:22:10.092762 2145 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:10.093262 2139 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260924 00:22:10.107822 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 46597.2us
W20260924 00:22:10.100981 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 79085.4us
I20260924 00:22:10.090075 1987 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260924 00:22:10.102387 3338 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:10.092399 3391 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:10.111344 2649 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:10.093875 3423 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:10.095988 3534 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260924 00:22:10.118625 3476 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:10.101358 3477 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.112306 1987 pending_rounds.cc:76] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 2
I20260924 00:22:10.121287 3391 raft_consensus.cc:686] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:10.116385 3338 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.125880 2067 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:10.126327 3338 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.132241 2142 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:10.132282 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:10.116357 3392 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:10.116219 3426 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:10.127840 1987 pending_rounds.cc:97] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260924 00:22:10.138875 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.137362 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:10.142932 2139 raft_consensus.cc:2379] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:10.141217 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:10.144270 2067 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:10.143752 3300 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:10.146605 3253 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:10.147753 3380 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:10.124639 3391 consensus_queue.cc:237] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.147949 3393 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:10.150682 1995 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:10.151989 2109 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:10.119504 3473 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260924 00:22:10.150321 1987 raft_consensus.cc:2866] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:10.153200 3392 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:10.084045 2138 rpcz_store.cc:276] 0924 00:22:08.789859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:08.790981 (+ 1122us) service_pool.cc:227] Handling call
0924 00:22:08.832920 (+ 41939us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:09.825093 (+992173us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:10.062385 (+237292us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b54000e06d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:10.062610 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2234927360}
I20260924 00:22:10.175302 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:10.176069 2065 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:10.177376 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:10.177649 2062 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260924 00:22:10.154030 3392 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.159659 3380 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:10.142124 3426 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.160280 3253 consensus_queue.cc:1225] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 8c129c01e1c047628f6a87b001aa7258 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260924 00:22:10.160684 3478 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:10.161445 3452 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.134227 3479 consensus_queue.cc:237] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:10.137496 3481 consensus_queue.cc:1046] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260924 00:22:10.135766 3498 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260924 00:22:10.139547 3471 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:10.167698 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:10.168442 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.222453 1993 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:10.169180 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.137064 3480 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:10.169469 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.224653 1992 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:10.170675 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.171957 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.172662 1995 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:10.122648 3311 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.193387 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.198433 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:10.154668 3300 raft_consensus.cc:686] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:10.207675 2649 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:10.175731 2640 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:10.209635 3534 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.007s
I20260924 00:22:10.173588 3473 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260924 00:22:10.230783 1994 raft_consensus.cc:2354] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:10.248764 3465 raft_consensus.cc:2741] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260924 00:22:10.224114 3471 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.202756 3478 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:10.220449 3489 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260924 00:22:10.302745 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:10.303404 3040 consensus_queue.cc:1046] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260924 00:22:10.303666 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 80276.7us
I20260924 00:22:10.307399 1996 raft_consensus.cc:2354] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3.
I20260924 00:22:10.308575 3311 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.308990 3536 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:10.305604 3452 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:10.309579 3536 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.309798 3452 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.249573 3498 consensus_queue.cc:1225] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 8c129c01e1c047628f6a87b001aa7258 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260924 00:22:10.317360 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:10.317862 3536 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:10.237416 3521 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.321055 2109 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:10.319792 2064 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:10.253419 3312 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.261417 3478 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:10.332122 3426 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.337232 2141 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:10.286375 3471 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:10.349011 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.338333 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:10.342478 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:10.285006 3414 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.346239 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:10.346469 3040 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:10.353705 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:10.260656 3454 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260924 00:22:10.355376 2143 raft_consensus.cc:2396] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:10.355939 3040 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:10.332623 3532 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:10.366523 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:10.336592 3452 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.369011 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:10.364269 3426 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:10.380322 2142 raft_consensus.cc:595] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:10.383378 2143 raft_consensus.cc:2354] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:10.351903 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:10.383332 2142 raft_consensus.cc:603] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:10.387796 2144 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:10.384541 3532 raft_consensus.cc:686] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:10.390760 3521 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:10.392256 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 426029us
I20260924 00:22:10.351313 3498 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260924 00:22:10.325075 3300 consensus_queue.cc:237] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.353511 3534 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260924 00:22:10.383147 3470 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:10.416117 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:10.417603 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:10.420713 2070 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:10.421293 2035 leader_election.cc:398] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:10.421593 2035 leader_election.cc:401] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:10.422243 1991 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:10.423444 3538 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:10.423868 3538 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:10.390947 3532 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:10.424428 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:10.392771 2136 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:10.445508 3452 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:10.447953 3481 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:10.451319 2066 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:10.460660 2109 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:10.388018 3426 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.416515 1988 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:10.411377 3465 consensus_queue.cc:1035] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260924 00:22:10.447119 3462 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:10.424666 3538 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:10.503865 3557 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260924 00:22:10.504132 3338 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:10.448341 3481 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:10.519797 2066 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:10.528172 2136 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:10.499787 3500 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
W20260924 00:22:10.530361 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 69774.1us
I20260924 00:22:10.496754 3471 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:10.537920 2110 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:10.454545 3473 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260924 00:22:10.548391 3426 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:10.534322 3536 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:10.536495 3338 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:10.549775 3481 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:10.564589 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 200208us
I20260924 00:22:10.580691 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:10.614288 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:10.559135 3452 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:10.558946 3500 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:10.588222 3475 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:10.597040 3476 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
I20260924 00:22:10.664160 3376 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 00:22:10.820235 3560 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20260924 00:22:10.738230 3376 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:10.850822 3559 log.cc:924] Time spent T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.188s user 0.000s sys 0.000s
I20260924 00:22:10.849642 3376 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:10.729302 3338 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.928012 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:10.928676 2145 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:10.939179 1991 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:10.901660 3338 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:10.817785 3500 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:10.954326 3040 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:10.929543 3481 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:10.953032 3538 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:10.941404 3475 raft_consensus.cc:686] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:22:10.948099 3548 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.329s user 0.019s sys 0.011s
I20260924 00:22:10.942049 3414 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:10.944046 3536 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:22:10.993333 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 60362.3us
W20260924 00:22:11.024123 3572 log.cc:924] Time spent T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260924 00:22:11.050617 2137 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.015s sys 0.072s
W20260924 00:22:11.056411 2137 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s user 0.016s sys 0.072s
I20260924 00:22:11.058712 3536 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:11.063768 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 137388us
I20260924 00:22:11.077085 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:11.082405 2036 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:11.086529 2142 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260924 00:22:11.087790 2036 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:11.096071 2141 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260924 00:22:11.101343 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 66912.5us
I20260924 00:22:11.106571 2068 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:11.108608 2036 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:11.126231 2143 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260924 00:22:11.082970 3380 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:11.105706 3311 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:11.101099 3507 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:11.083496 3470 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:11.054214 3475 consensus_queue.cc:237] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:11.144511 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 90881.5us
I20260924 00:22:11.179943 1993 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260924 00:22:11.176839 3581 log.cc:924] Time spent T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
W20260924 00:22:11.189517 3574 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
W20260924 00:22:11.190197 3575 log.cc:924] Time spent T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.200s user 0.000s sys 0.000s
I20260924 00:22:11.194525 2614 leader_election.cc:398] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3.
I20260924 00:22:11.194882 2614 leader_election.cc:401] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260924 00:22:11.196465 3550 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3.
W20260924 00:22:11.199575 3478 consensus_meta.cc:287] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.849s user 0.008s sys 0.003s
I20260924 00:22:11.200007 3478 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.170236 3468 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:11.175375 3311 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:11.122701 3514 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260924 00:22:11.209012 3414 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:11.213958 2136 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:11.219079 3441 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
W20260924 00:22:11.223316 3563 log.cc:924] Time spent T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.165s user 0.001s sys 0.000s
I20260924 00:22:11.185374 3470 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:11.231817 3565 log.cc:924] Time spent T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.421s user 0.000s sys 0.014s
I20260924 00:22:11.232945 3471 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:11.189545 3577 log.cc:924] Time spent T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
W20260924 00:22:11.255673 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 147129us
I20260924 00:22:11.259895 2036 leader_election.cc:398] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:11.263748 2036 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:11.270530 3514 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:11.273270 3376 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.276047 3376 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:11.250650 3441 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:11.227258 3311 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:11.304893 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:11.325389 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 219570us
I20260924 00:22:11.363582 3040 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:11.364246 3040 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.258311 3471 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:11.209990 3550 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260924 00:22:11.370965 1992 raft_consensus.cc:3032] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:11.375985 1992 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:11.373811 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.377179 1994 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:11.377271 3040 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:11.250741 3414 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.227061 3468 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:11.298930 3441 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.383096 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:11.376140 3429 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260924 00:22:11.388919 1991 raft_consensus.cc:2354] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:11.395605 1992 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.399262 1992 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:11.439728 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:11.441210 2036 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:11.441743 1992 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:11.438632 3478 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:11.438860 3507 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:11.398519 3441 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:11.394999 3391 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:11.461300 3552 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:11.397858 3534 consensus_queue.cc:1046] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260924 00:22:11.443607 3507 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.516912 2140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1092) took 1335 ms. Trace:
I20260924 00:22:11.452929 3421 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:11.465906 3423 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260924 00:22:11.466821 3253 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260924 00:22:11.532440 1991 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:11.535779 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.537875 3473 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:11.540385 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:11.541208 2144 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:11.537966 3376 raft_consensus.cc:3032] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:11.520602 2140 rpcz_store.cc:276] 0924 00:22:10.181889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:10.181989 (+ 100us) service_pool.cc:227] Handling call
0924 00:22:10.191396 (+ 9407us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:10.285652 (+ 94256us) raft_consensus.cc:1520] Early marking committed up to index 1
0924 00:22:10.285690 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:10.286073 (+ 383us) log.cc:842] Serialized 26 byte log entry
0924 00:22:10.462406 (+176333us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:10.462449 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:10.663706 (+201257us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b54000e0bd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:10.663772 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:11.225605 (+561833us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:11.516655 (+291050us) raft_consensus.cc:1671] finished
0924 00:22:11.516677 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:11.516846 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":245039744,"thread_start_us":175992,"threads_started":1,"wal-append.queue_time_us":365060}
I20260924 00:22:11.528537 3423 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:11.514979 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:11.534793 3538 consensus_meta.cc:287] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.573s user 0.011s sys 0.000s
I20260924 00:22:11.558231 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:11.558236 3423 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:11.561841 3590 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260924 00:22:11.568522 3478 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:11.569583 3507 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:11.570807 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:11.532006 3500 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:11.513944 3552 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.539467 3452 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:11.539367 3421 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:11.468847 3391 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.549404 3376 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:11.579209 1987 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:11.594244 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.595335 1994 raft_consensus.cc:2396] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260924 00:22:11.596676 1988 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:11.549642 3473 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:11.604471 2069 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.009s sys 0.014s
W20260924 00:22:11.605441 2069 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s user 0.010s sys 0.014s
I20260924 00:22:11.621640 2137 raft_consensus.cc:1261] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:11.624745 3380 raft_consensus.cc:979] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:11.629446 3429 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260924 00:22:11.630658 3429 tablet_bootstrap.cc:492] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:11.631677 3429 ts_tablet_manager.cc:1397] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 1.810s user 0.039s sys 0.004s
I20260924 00:22:11.597218 3414 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:11.615417 3424 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.502s user 0.005s sys 0.017s
I20260924 00:22:11.585244 3421 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.593358 3479 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260924 00:22:11.645853 3532 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:11.677273 3552 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:11.720263 2063 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.010s sys 0.006s
I20260924 00:22:11.649302 3392 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:11.683373 3380 raft_consensus.cc:1067] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
W20260924 00:22:11.733251 2071 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.008s sys 0.007s
W20260924 00:22:11.770886 2071 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.010s sys 0.009s
W20260924 00:22:11.748965 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 307007us
W20260924 00:22:11.750936 2063 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.010s sys 0.009s
W20260924 00:22:11.722178 1989 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.024s sys 0.019s
W20260924 00:22:11.772709 1989 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.024s sys 0.019s
I20260924 00:22:11.649343 3423 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.721586 3532 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.763900 3585 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:11.680068 3534 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20260924 00:22:11.633229 3376 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.773890 2036 leader_election.cc:398] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3.
I20260924 00:22:11.748687 3552 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:11.778612 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.760731 3507 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:11.767869 3514 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:11.771330 3441 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:11.783046 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.789413 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.790061 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.774564 3421 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:11.800057 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 418314us
I20260924 00:22:11.815028 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:11.816275 1994 raft_consensus.cc:2396] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260924 00:22:11.769727 3392 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:11.859937 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 55777us
I20260924 00:22:11.782738 3585 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:11.790966 3376 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:11.823276 3475 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.011s sys 0.013s
I20260924 00:22:11.884413 3421 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260924 00:22:11.794179 3391 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:11.782976 3552 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.881959 3585 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:11.881916 3380 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:11.942656 3475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.012s sys 0.013s
I20260924 00:22:11.794301 3507 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.795387 3441 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:11.791098 3514 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:11.962350 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 70879.7us
W20260924 00:22:11.960317 3600 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.228s user 0.001s sys 0.000s
I20260924 00:22:11.945168 3473 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:11.941988 3482 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:11.962301 3353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.022s sys 0.027s
I20260924 00:22:11.967034 3380 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.004678 1991 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
W20260924 00:22:11.998315 3353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.022s sys 0.027s
I20260924 00:22:11.997717 3423 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.004423 3424 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.028402 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:12.003914 3392 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:12.021340 3612 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:12.032460 3597 log.cc:924] Time spent T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.249s user 0.000s sys 0.001s
I20260924 00:22:12.035288 3391 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:12.038738 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:12.091948 1992 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:12.002439 3380 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:11.999197 3610 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:12.002575 3481 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:12.000316 3482 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.161964 3552 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.163872 3529 consensus_queue.cc:1046] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:12.166551 3514 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.171345 3380 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.172011 3380 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.178030 3441 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.090456 3423 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.179126 3423 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.138618 3391 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:12.188439 3311 consensus_meta.cc:287] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.822s user 0.005s sys 0.005s
I20260924 00:22:12.190894 3468 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:12.153792 3481 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:12.278837 2035 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 1.911s (SENT) [suppressed 4 similar messages]
I20260924 00:22:12.318358 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1018) took 1226 ms. Trace:
I20260924 00:22:12.189019 3311 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.318820 2071 rpcz_store.cc:276] 0924 00:22:11.091598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.091784 (+ 186us) service_pool.cc:227] Handling call
0924 00:22:11.118851 (+ 27067us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 00:22:11.127791 (+ 8940us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:11.127831 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 00:22:11.147714 (+ 19883us) log.cc:842] Serialized 52 byte log entry
0924 00:22:11.514327 (+366613us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b4800210258. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d99637f1 00007fb6d995a8e7 00007fb6d995a77a 00007fb6d9a4e95f 00007fb6d9a4e627 00007fb6d9a62b2c 00007fb6d9a62ad8 00007fb6d9a62a58 00007fb6d9a62a13 00007fb6d9a61cbc 00007fb6d89e3be6
0924 00:22:11.771423 (+257096us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:11.771477 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:11.771566 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:12.295137 (+523571us) raft_consensus.cc:1671] finished
0924 00:22:12.295161 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:12.318293 (+ 23132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8478,"raft.queue_time_us":28694,"raft.run_cpu_time_us":3396,"raft.run_wall_time_us":347000,"spinlock_wait_cycles":8248704,"thread_start_us":569732,"threads_started":1,"wal-append.queue_time_us":778443}
W20260924 00:22:12.321697 3596 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.357s user 0.000s sys 0.001s
I20260924 00:22:12.323067 2142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1096) took 1229 ms. Trace:
I20260924 00:22:12.323408 2142 rpcz_store.cc:276] 0924 00:22:11.094195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.094329 (+ 134us) service_pool.cc:227] Handling call
0924 00:22:11.094677 (+ 348us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:11.105920 (+ 11243us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:11.105953 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:11.106417 (+ 464us) log.cc:842] Serialized 26 byte log entry
0924 00:22:11.590799 (+484382us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:11.590849 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:11.590935 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:12.322863 (+731928us) raft_consensus.cc:1671] finished
0924 00:22:12.322890 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:12.323015 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18272,"raft.queue_time_us":26539,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":171,"thread_start_us":484056,"threads_started":1,"wal-append.queue_time_us":583163}
I20260924 00:22:12.323307 3311 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:12.340533 3471 consensus_meta.cc:287] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.954s user 0.008s sys 0.004s
I20260924 00:22:12.295128 3532 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:12.312707 3468 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.349236 3468 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.352314 3468 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.205711 3610 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.043721 3424 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.359565 3424 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:12.343101 3482 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:12.152417 3475 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:12.296330 3376 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.204493 3429 raft_consensus.cc:348] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf [term 4 NON_PARTICIPANT]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.145810 3612 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
W20260924 00:22:12.362934 1987 consensus_meta.cc:287] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.760s user 0.009s sys 0.001s
I20260924 00:22:12.401641 3423 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.402010 3612 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:22:12.408381 3481 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.409996 3429 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
W20260924 00:22:12.410673 3605 log.cc:924] Time spent T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.351s user 0.000s sys 0.001s
I20260924 00:22:12.402314 3380 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.439272 2066 raft_consensus.cc:1261] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:12.439396 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 156) took 1502 ms. Trace:
I20260924 00:22:12.439918 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1028) took 1312 ms. Trace:
I20260924 00:22:12.440209 3380 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:12.440775 3380 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.439970 1989 rpcz_store.cc:276] 0924 00:22:10.936944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:10.940660 (+ 3716us) service_pool.cc:227] Handling call
0924 00:22:10.968499 (+ 27839us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:11.015585 (+ 47086us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:11.021011 (+ 5426us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:11.069734 (+ 48723us) log.cc:842] Serialized 26 byte log entry
0924 00:22:11.774527 (+704793us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:11.775962 (+ 1435us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:11.777429 (+ 1467us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:12.439098 (+661669us) raft_consensus.cc:1671] finished
0924 00:22:12.439140 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:12.439323 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":34831,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":18527,"thread_start_us":647130,"threads_started":1,"wal-append.queue_time_us":857968}
I20260924 00:22:12.442983 3380 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.379136 3376 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.444813 3376 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.381524 3482 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.429563 3353 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:12.445456 3482 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.445591 3353 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.447405 3482 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.447897 3353 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:12.430037 3500 consensus_meta.cc:287] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.856s user 0.009s sys 0.002s
I20260924 00:22:12.392731 3610 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.453553 3424 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.436388 3429 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:12.440335 2066 rpcz_store.cc:276] 0924 00:22:11.127704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:12.438620 (+1310916us) service_pool.cc:227] Handling call
0924 00:22:12.438932 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:12.439692 (+ 760us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:12.439868 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:12.456183 2641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45594 (request call id 298) took 1075 ms. Trace:
I20260924 00:22:12.444844 1893 catalog_manager.cc:5495] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:12.354526 3471 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.454000 3424 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.379509 3475 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.450251 1987 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:12.568565 3610 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:12.568719 3429 ts_tablet_manager.cc:1428] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.936s user 0.010s sys 0.004s
I20260924 00:22:12.569259 3423 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:12.569473 3612 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:12.570291 2066 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20260924 00:22:12.570815 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 112126us
I20260924 00:22:12.580119 2070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:22:12.580787 2070 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:12.572103 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:12.581199 2070 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:12.581530 2063 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:12.496776 1893 catalog_manager.cc:5495] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 4, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
W20260924 00:22:12.582990 2614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.974s user 0.017s sys 0.082s
W20260924 00:22:12.583297 2614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.974s user 0.017s sys 0.082s
I20260924 00:22:12.583791 1893 catalog_manager.cc:5495] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:22:12.572687 3338 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.043s sys 0.055s
I20260924 00:22:12.453912 3585 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.616075 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.713121 3471 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:12.713729 3423 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.713359 2067 raft_consensus.cc:2379] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:12.617404 1893 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:22:12.638231 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 55746.4us
I20260924 00:22:12.717819 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.718396 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.718677 2064 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
W20260924 00:22:12.577764 3429 ts_tablet_manager.cc:1460] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Tablet startup took 945ms
I20260924 00:22:12.574575 3521 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:12.731483 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.717526 1893 catalog_manager.cc:5495] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 2 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:12.734146 2064 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
W20260924 00:22:12.719167 3338 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.038s user 0.044s sys 0.063s
I20260924 00:22:12.456634 2641 rpcz_store.cc:276] 0924 00:22:11.380860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.382572 (+ 1712us) service_pool.cc:227] Handling call
0924 00:22:12.454392 (+1071820us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b5800005a98. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:12.455526 (+ 1134us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:12.455759 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:12.455797 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:12.455851 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:12.455894 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:12.455965 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:12.456109 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52130688}
I20260924 00:22:12.747560 3594 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:12.747597 3468 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:12.747704 2615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.107s user 0.115s sys 0.209s
W20260924 00:22:12.751861 1995 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.984s user 0.007s sys 0.060s
I20260924 00:22:12.583405 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1449) took 1281 ms. Trace:
W20260924 00:22:12.764395 2069 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.116s user 0.020s sys 0.043s
I20260924 00:22:12.579203 3480 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:12.769510 1995 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.002s user 0.008s sys 0.061s
W20260924 00:22:12.770071 2069 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.127s user 0.021s sys 0.043s
W20260924 00:22:12.771306 2615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.151s user 0.115s sys 0.210s
I20260924 00:22:12.687675 3481 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:12.781898 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.742841 3534 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:12.789749 2141 raft_consensus.cc:1226] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:12.786088 2066 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:12.689556 3612 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.797433 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.780866 2063 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:12.835469 2066 raft_consensus.cc:2396] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260924 00:22:12.790766 1990 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.027s user 0.022s sys 0.080s
I20260924 00:22:12.840579 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:12.835054 3376 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:12.842731 3376 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.844681 3376 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.845925 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:12.579861 3610 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.749778 3521 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.847115 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.858345 2066 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:12.768792 2070 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:12.856292 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:12.856600 2648 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:12.774566 1987 rpcz_store.cc:276] 0924 00:22:11.301612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.411054 (+109442us) service_pool.cc:227] Handling call
0924 00:22:12.579437 (+1168383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:12.829430 3353 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:12.760083 3468 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.787184 3480 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.865155 2070 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.837862 3210 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:12.870616 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:12.860649 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:12.872890 3569 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:12.873338 2108 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.019s sys 0.054s
W20260924 00:22:12.888428 2108 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.128s user 0.019s sys 0.054s
I20260924 00:22:12.874199 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1162) took 1792 ms. Trace:
I20260924 00:22:12.874830 3610 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.752676 3594 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.789043 3481 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:12.854816 1990 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.088s user 0.022s sys 0.080s
I20260924 00:22:12.863132 3423 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:12.882120 3585 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:12.868818 3353 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:12.884374 3521 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:12.886973 3468 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:12.879925 2066 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:12.897408 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:12.796036 3534 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:13.101801 2649 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.136s user 0.007s sys 0.011s
W20260924 00:22:13.095561 1996 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.028s sys 0.030s
I20260924 00:22:13.115136 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.126873 2142 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:13.074626 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.128449 2070 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:12.896027 3471 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.122383 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:13.126348 1996 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.192s user 0.029s sys 0.031s
W20260924 00:22:13.107659 2649 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.142s user 0.008s sys 0.011s
I20260924 00:22:13.133917 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.134706 2139 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:12.924830 3587 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:13.139513 3601 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:13.140065 3601 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.140800 3612 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.140858 3476 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:13.141497 3476 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.142410 3601 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.013149 3475 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.005851 3584 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:13.143525 2649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 288) took 1729 ms. Trace:
I20260924 00:22:13.011482 3481 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.079434 3480 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:12.891986 3210 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.009704 3532 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:12.924391 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:12.895830 3486 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:12.925604 3618 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:12.925447 3588 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:13.146960 2035 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.003s sys 0.031s
I20260924 00:22:13.181254 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.182770 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.177871 2649 rpcz_store.cc:276] 0924 00:22:11.414281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.414436 (+ 155us) service_pool.cc:227] Handling call
0924 00:22:11.805802 (+391366us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b5800005a98. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:11.806048 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:11.806302 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 6
0924 00:22:11.897466 (+ 91164us) log.cc:842] Serialized 14 byte log entry
0924 00:22:13.143046 (+1245580us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:13.143231 (+ 185us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:22:13.143279 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:13.143342 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:13.143452 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7978,"prepare.queue_time_us":91148,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":2397,"raft.queue_time_us":9577,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"spinlock_wait_cycles":4823424,"thread_start_us":1166791,"threads_started":1}
I20260924 00:22:12.892776 1988 rpcz_store.cc:276] 0924 00:22:11.082192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.082618 (+ 426us) service_pool.cc:227] Handling call
0924 00:22:11.171129 (+ 88511us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:11.600070 (+428941us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:12.873863 (+1273793us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54000753d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:12.874111 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":939054976}
I20260924 00:22:13.199919 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:13.183367 3393 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.143s user 0.013s sys 0.026s
I20260924 00:22:13.267212 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:13.267809 2140 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:13.146905 3595 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.993s user 0.013s sys 0.022s
I20260924 00:22:13.265010 2139 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:12.966830 3569 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:13.265082 2035 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.951s user 0.003s sys 0.039s
I20260924 00:22:13.176954 3601 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:13.265738 2142 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
W20260924 00:22:13.266809 2034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.468s user 0.015s sys 0.091s
W20260924 00:22:13.340378 2034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.541s user 0.015s sys 0.091s
W20260924 00:22:13.275731 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 996847us
W20260924 00:22:13.265010 3605 log.cc:924] Time spent T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260924 00:22:13.197718 3611 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:13.179714 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.132864 3471 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:13.274839 2145 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.274s user 0.000s sys 0.024s
I20260924 00:22:13.346647 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:13.270406 3393 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.227s user 0.017s sys 0.027s
I20260924 00:22:13.361550 3594 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.362092 3210 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.362838 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.364033 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.364415 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.365214 2067 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:13.365553 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.366104 2068 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:13.367043 3353 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.198850 3532 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.369377 3532 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:13.369678 3621 log.cc:924] Time spent T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.226s user 0.000s sys 0.000s
I20260924 00:22:13.371763 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:13.372792 2070 raft_consensus.cc:2379] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
W20260924 00:22:13.342819 2110 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.321s user 0.011s sys 0.013s
I20260924 00:22:13.180547 3587 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.258642 3468 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:13.258556 3439 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:13.348372 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:13.379320 3439 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:12.733392 3429 ts_tablet_manager.cc:1462] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Trace:
0924 00:22:07.087245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:07.090868 (+ 3623us) service_pool.cc:227] Handling call
0924 00:22:07.173017 (+ 82149us) ts_tablet_manager.cc:928] T d2ef9fa9e5dd48218a5c243de0c0e49a P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
0924 00:22:07.690667 (+517650us) tablet_copy_client.cc:361] Tablet copy session begun
0924 00:22:08.296560 (+605893us) tablet_metadata.cc:641] Metadata flushed
0924 00:22:08.296828 (+ 268us) tablet_copy_client.cc:490] Wrote new tablet metadata
0924 00:22:08.710468 (+413640us) tablet_copy_client.cc:411] Wrote new consensus metadata
0924 00:22:08.710794 (+ 326us) tablet_replica.cc:205] Creating consensus instance
0924 00:22:08.711154 (+ 360us) ts_tablet_manager.cc:983] Replica registered.
0924 00:22:08.711221 (+ 67us) inbound_call.cc:173] Queueing success response
0924 00:22:09.060665 (+349444us) spinlock_profiling.cc:243] Waited 6.85 ms on lock 0x7b18002c2ae0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d1b0b4ca 00007fb6db1ecf99 00007fb6d19cf628 00007fb6d1b94002 00007fb6d99b1213 00007fb6d99b0a4d 00007fb6d9a49a17 00007fb6d9a7aa8e 00007fb6d9a605ad
0924 00:22:09.821944 (+761279us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0924 00:22:11.632669 (+1810725us) ts_tablet_manager.cc:1429] Starting tablet replica
0924 00:22:11.992613 (+359944us) tablet_replica.cc:259] Starting instrumentation
0924 00:22:12.202529 (+209916us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":340,"mutex_wait_us":1151,"raft.queue_time_us":113256,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":8553,"spinlock_wait_cycles":15755520,"tablet-copy.queue_time_us":78087,"tablet-download-pool.queue_time_us":3900,"tablet-download-pool.run_cpu_time_us":1960,"tablet-download-pool.run_wall_time_us":355979,"thread_start_us":108800,"threads_started":2}
I20260924 00:22:13.381652 3439 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.199853 3486 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.383597 3587 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.384348 3468 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.199720 3584 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.394057 3584 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.370359 3604 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:13.274829 3595 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.132s user 0.016s sys 0.024s
W20260924 00:22:13.399654 2069 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.009s sys 0.052s
W20260924 00:22:13.399961 2069 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.009s sys 0.052s
W20260924 00:22:13.400799 2071 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.830s user 0.021s sys 0.064s
I20260924 00:22:13.384881 3486 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.387583 3476 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.388629 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1176) took 1608 ms. Trace:
I20260924 00:22:13.390821 2649 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:13.270299 3618 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:13.397608 3338 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.012s sys 0.048s
W20260924 00:22:13.355772 2145 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.355s user 0.000s sys 0.024s
W20260924 00:22:13.439957 3338 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s user 0.012s sys 0.049s
I20260924 00:22:13.357493 2139 raft_consensus.cc:2379] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
W20260924 00:22:13.375164 2110 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.354s user 0.012s sys 0.013s
I20260924 00:22:13.403535 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1184) took 2839 ms. Trace:
I20260924 00:22:13.360893 3611 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:13.409400 1893 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.029s sys 0.056s
I20260924 00:22:13.345106 3588 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:13.442301 1893 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.659s user 0.030s sys 0.056s
I20260924 00:22:13.443445 3611 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.360461 3601 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.441978 2069 rpcz_store.cc:276] 0924 00:22:10.561399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:10.566464 (+ 5065us) service_pool.cc:227] Handling call
0924 00:22:10.567816 (+ 1352us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 00:22:10.929777 (+361961us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:10.929808 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 00:22:10.943448 (+ 13640us) log.cc:842] Serialized 52 byte log entry
0924 00:22:11.605882 (+662434us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:12.775638 (+1169756us) log.cc:842] Serialized 14 byte log entry
0924 00:22:13.400762 (+625124us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:13.400901 (+ 139us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:13.400940 (+ 39us) raft_consensus.cc:1671] finished
0924 00:22:13.400950 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:13.401076 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34834,"prepare.queue_time_us":1488285,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":365567,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":1464,"thread_start_us":2392082,"threads_started":3,"wal-append.queue_time_us":733478,"wal-append.run_cpu_time_us":10755,"wal-append.run_wall_time_us":1039156}
I20260924 00:22:13.445319 3601 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:13.441979 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 1.42071e+06us
I20260924 00:22:13.448169 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.450802 3618 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.435447 1996 rpcz_store.cc:276] 0924 00:22:11.779727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.779877 (+ 150us) service_pool.cc:227] Handling call
0924 00:22:11.789315 (+ 9438us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:11.789618 (+ 303us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:11.789643 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:11.790029 (+ 386us) log.cc:842] Serialized 26 byte log entry
0924 00:22:13.132281 (+1342252us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:13.132335 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:13.132421 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:13.388393 (+255972us) raft_consensus.cc:1671] finished
0924 00:22:13.388417 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:13.388564 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12046,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":77287,"thread_start_us":1197094,"threads_started":1,"wal-append.queue_time_us":1519008}
W20260924 00:22:13.404284 3500 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.003s sys 0.008s
W20260924 00:22:13.419718 1961 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.006s sys 0.007s
I20260924 00:22:13.401870 3612 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:13.433959 2071 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.863s user 0.022s sys 0.064s
I20260924 00:22:13.403995 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1169) took 1815 ms. Trace:
I20260924 00:22:13.405364 3423 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.443555 3588 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.395483 3604 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.455097 3569 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.411827 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:22:13.453750 3500 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s user 0.004s sys 0.008s
I20260924 00:22:13.451583 2138 raft_consensus.cc:595] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:13.434435 3534 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:13.453500 1961 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.006s sys 0.007s
I20260924 00:22:13.461386 3624 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.476639 2138 raft_consensus.cc:603] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:13.485673 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.455727 3612 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.462993 3423 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.477424 2641 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:13.498909 3604 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.485622 3624 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.500355 3391 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.501039 3391 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.501426 3624 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.501963 3624 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.501663 3391 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.502424 3624 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.504043 3391 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.504302 3624 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.504444 3468 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.506441 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.507181 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.507622 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.507800 2142 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:13.509357 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.522622 2140 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:13.510097 2139 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:13.534060 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:13.543179 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 269021us
I20260924 00:22:13.528861 3471 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:13.544858 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 195165us
I20260924 00:22:13.545914 3505 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:13.546224 3624 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.546487 3505 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.546782 3624 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.554013 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.554852 2139 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:13.549088 3624 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.561895 3414 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:13.558228 3470 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.558056 3391 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.542100 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.548957 3505 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.573450 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.578259 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1111) took 1597 ms. Trace:
I20260924 00:22:13.579545 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:13.579721 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.578645 2145 rpcz_store.cc:276] 0924 00:22:11.980896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.991424 (+ 10528us) service_pool.cc:227] Handling call
0924 00:22:11.991737 (+ 313us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:13.440142 (+1448405us) raft_consensus.cc:1520] Early marking committed up to index 1
0924 00:22:13.440179 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:13.440635 (+ 456us) log.cc:842] Serialized 26 byte log entry
0924 00:22:13.576806 (+136171us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:13.576860 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:13.576948 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:13.577005 (+ 57us) raft_consensus.cc:1671] finished
0924 00:22:13.577023 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:13.578197 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":140712,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":13763,"thread_start_us":1583773,"threads_started":2,"wal-append.queue_time_us":132755}
I20260924 00:22:13.580833 2144 raft_consensus.cc:2379] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:13.557631 3471 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.582729 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:13.580796 2111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.463s user 0.011s sys 0.013s
W20260924 00:22:13.586601 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.576s user 0.012s sys 0.014s
I20260924 00:22:13.592224 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:13.563978 3391 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:13.598106 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 50499.1us
I20260924 00:22:13.577956 2137 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:13.600203 2035 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:13.588359 3471 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.609570 3423 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.610126 3471 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.599946 3520 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:13.610997 3520 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:13.611373 3520 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:13.612473 2145 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:13.616184 2035 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:13.597078 3470 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.617537 3470 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.619441 3470 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.622053 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-d2ef9fa9e5dd48218a5c243de0c0e49a received from {username='slave'} at 127.0.0.1:39164
I20260924 00:22:13.622494 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-d2ef9fa9e5dd48218a5c243de0c0e49a on tablet d2ef9fa9e5dd48218a5c243de0c0e49a with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:13.622745 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.621918 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.626294 3391 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.627015 3521 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.627554 3521 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.597122 3486 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260924 00:22:13.455615 1990 rpcz_store.cc:276] 0924 00:22:11.588372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.590870 (+ 2498us) service_pool.cc:227] Handling call
0924 00:22:11.618044 (+ 27174us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:11.720821 (+102777us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:11.723589 (+ 2768us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:11.762981 (+ 39392us) log.cc:842] Serialized 26 byte log entry
0924 00:22:13.029100 (+1266119us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:13.046892 (+ 17792us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:13.054013 (+ 7121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:13.388078 (+334065us) raft_consensus.cc:1671] finished
0924 00:22:13.396390 (+ 8312us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:13.402124 (+ 5734us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40791,"raft.queue_time_us":13741,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":29516,"thread_start_us":1159735,"threads_started":1,"wal-append.queue_time_us":1332022}
I20260924 00:22:13.631249 3424 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.632851 3653 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:13.634471 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.631551 3505 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:13.641119 3606 log.cc:924] Time spent T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.845s user 0.000s sys 0.001s
I20260924 00:22:13.645893 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.647929 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.640578 3505 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.654667 3507 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.657387 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1110) took 1879 ms. Trace:
I20260924 00:22:13.657768 2141 rpcz_store.cc:276] 0924 00:22:11.777617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.777779 (+ 162us) service_pool.cc:227] Handling call
0924 00:22:11.778137 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:12.790379 (+1012242us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:13.657115 (+866736us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54000e0bd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:13.657331 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1852227968}
I20260924 00:22:13.659155 3505 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.659245 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:13.661068 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 44959.4us
I20260924 00:22:13.661526 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.661917 2035 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:13.647837 3521 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.636425 3653 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.674041 3653 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:13.675719 1942 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 8c129c01e1c047628f6a87b001aa7258 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:13.663455 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.667517 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:22:13.686977 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet aa5e2f99029440bd9d42ec2a01200d4a, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:13.687265 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet aa5e2f99029440bd9d42ec2a01200d4a, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:13.697242 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 24240f07434f49e9bef897dac3fbe801, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:13.697582 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 24240f07434f49e9bef897dac3fbe801, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:13.700067 3614 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 1.134s user 0.003s sys 0.017s
W20260924 00:22:13.701159 3607 log.cc:924] Time spent T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 1.734s user 0.005s sys 0.019s
W20260924 00:22:13.702435 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c4982c422b5247e29807c869b18b7eb8, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:13.702881 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c4982c422b5247e29807c869b18b7eb8, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.706069 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 1456) took 2334 ms. Trace:
I20260924 00:22:13.706435 1995 rpcz_store.cc:276] 0924 00:22:11.371721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:11.372262 (+ 541us) service_pool.cc:227] Handling call
0924 00:22:11.372628 (+ 366us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:11.373591 (+ 963us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:11.373625 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:11.380004 (+ 6379us) log.cc:842] Serialized 26 byte log entry
0924 00:22:11.720031 (+340027us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:12.775225 (+1055194us) log.cc:842] Serialized 14 byte log entry
0924 00:22:12.793894 (+ 18669us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:12.793982 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:13.698310 (+904328us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:13.705735 (+ 7425us) raft_consensus.cc:1671] finished
0924 00:22:13.705850 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:13.705987 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22075,"prepare.queue_time_us":80458,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":21229,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":38716,"spinlock_wait_cycles":7039360,"thread_start_us":1347213,"threads_started":2,"wal-append.queue_time_us":505100}
I20260924 00:22:13.704100 3520 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.713538 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:13.714241 2640 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:13.713981 3520 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.718990 3481 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:13.719544 3481 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:13.729295 3380 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:13.729871 3380 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.730914 3040 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:13.731383 3040 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.733371 3594 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:13.733870 3594 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.735267 1996 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260924 00:22:13.735611 1996 pending_rounds.cc:76] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 1
I20260924 00:22:13.735649 3594 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.735917 1996 pending_rounds.cc:97] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260924 00:22:13.736230 1996 raft_consensus.cc:2866] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:13.737234 3520 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:13.737820 3520 consensus_queue.cc:1225] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260924 00:22:13.738874 1891 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:13.740433 1891 catalog_manager.cc:5495] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 3 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:13.741839 3534 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:13.742379 3534 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.742264 1891 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:13.744113 3612 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.744335 3534 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.743795 1891 catalog_manager.cc:5495] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 3, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 3 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:13.746253 3423 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:13.746624 3423 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.746707 3500 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:13.747006 3423 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:13.747287 3500 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:13.747788 2640 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:13.748950 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:13.713565 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bf0af13699974118a86f602c1fa9490b, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
W20260924 00:22:13.750560 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet bf0af13699974118a86f602c1fa9490b, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:13.750766 3391 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:13.751138 3391 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.751475 3391 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:13.753826 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1030) took 1185 ms. Trace:
I20260924 00:22:13.754707 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:13.756146 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 94314.8us
I20260924 00:22:13.754184 2071 rpcz_store.cc:276] 0924 00:22:12.568167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:12.569825 (+ 1658us) service_pool.cc:227] Handling call
0924 00:22:12.570073 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:12.570258 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 3
0924 00:22:13.454028 (+883770us) log.cc:842] Serialized 14 byte log entry
0924 00:22:13.637582 (+183554us) log.cc:842] Serialized 14 byte log entry
0924 00:22:13.753432 (+115850us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:13.753491 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:13.753546 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:13.753628 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:13.753766 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115602,"prepare.queue_time_us":564583,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":2565,"raft.queue_time_us":159102,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":1066251,"threads_started":3,"wal-append.queue_time_us":169933}
I20260924 00:22:13.758947 2035 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:13.733057 3040 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.760629 3654 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:13.761077 3653 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.761291 3654 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.761814 3654 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:13.761672 3653 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.763935 3653 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.760707 3642 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:13.771909 1995 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260924 00:22:13.772289 1995 pending_rounds.cc:76] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 2
I20260924 00:22:13.772483 1995 pending_rounds.cc:97] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260924 00:22:13.772738 1995 raft_consensus.cc:2866] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260924 00:22:13.764644 2036 connection.cc:232] RPC call timeout handler was delayed by -1.7039619909999146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
I20260924 00:22:13.731825 3380 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:13.788055 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:13.809304 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 50445.6us
I20260924 00:22:13.770931 3610 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:13.787112 3500 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:13.819607 2035 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:13.820060 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 90188.2us
I20260924 00:22:13.784219 3479 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:22:13.821719 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.813649 2063 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:13.823223 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:13.823784 2145 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:13.823324 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.824373 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.824918 2063 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:13.826869 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.827874 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.829458 2069 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260924 00:22:13.784605 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:13.831800 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.832479 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:13.832597 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:13.833171 2143 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:13.836272 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:13.836956 2063 raft_consensus.cc:2354] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3.
I20260924 00:22:13.836824 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:13.837452 1995 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:13.837965 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.838732 2069 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:13.840057 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:13.814626 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:13.867112 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.834513 1992 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:13.860239 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.868734 2069 raft_consensus.cc:2396] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260924 00:22:13.860235 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.870275 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.871618 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.785244 3642 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.861464 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:13.874014 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.874143 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.874773 2065 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:13.863138 2064 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:13.830888 3481 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:13.863994 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.863193 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.866149 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.878413 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.866477 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.866681 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.879151 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.879112 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:13.879863 2144 raft_consensus.cc:2379] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:13.817688 3649 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:13.881886 2068 raft_consensus.cc:1226] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:13.860889 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:13.833598 3507 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:13.884645 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.885340 2068 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:13.874102 2143 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:13.876715 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.884557 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.887550 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.877293 2070 raft_consensus.cc:2396] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260924 00:22:13.888132 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:13.877449 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.865180 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.858229 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:13.816217 3612 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.840224 3479 raft_consensus.cc:686] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:13.882726 2139 raft_consensus.cc:2362] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5. Re-sending same reply.
I20260924 00:22:13.892764 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.890520 2063 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:13.893548 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.883476 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.883494 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.818753 3610 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:13.888137 2064 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:13.891631 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.894351 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.895777 2067 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:13.880810 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:13.990202 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:13.944141 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.898588 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.001253 2139 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:13.983173 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.950533 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.003423 2035 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:14.003751 2070 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:13.934682 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:13.922168 2143 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:14.006286 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.910238 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.007678 1992 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260924 00:22:14.009675 2035 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:14.010874 2035 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:13.881740 3532 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.012563 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.013166 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:13.882035 3649 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:13.889217 3529 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.012616 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.934350 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:14.019225 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:13.890388 2640 raft_consensus.cc:1226] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:14.019706 2063 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:22:13.949558 3665 log.cc:924] Time spent T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260924 00:22:13.896224 3507 raft_consensus.cc:686] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:13.950126 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:14.022039 3649 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:14.018607 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.022365 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.022769 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.015419 3659 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:14.015465 3655 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:14.014163 3656 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:13.896736 3612 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.080358 3507 consensus_queue.cc:237] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.016952 2139 raft_consensus.cc:1226] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:14.044939 2070 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:13.934131 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:14.019042 3481 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:14.015308 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.046602 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:13.886291 3653 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20260924 00:22:14.084647 1992 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:14.043380 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.063241 3659 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.020517 3532 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.093950 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.091209 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.042622 3479 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:14.092062 2071 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:14.032701 3529 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.086839 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.091261 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.093562 2145 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:14.097968 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.101707 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.089048 3656 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:14.103052 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.094707 3659 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:14.106595 2067 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:14.109638 2141 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:14.096427 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.089156 3481 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.118222 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:14.111340 3656 raft_consensus.cc:686] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:14.124321 1992 pending_rounds.cc:76] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 1
I20260924 00:22:14.098935 3654 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.142746 1992 pending_rounds.cc:97] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260924 00:22:14.146473 1992 raft_consensus.cc:2866] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:14.119158 3610 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:14.181799 1891 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 3, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 3 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:14.194495 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.197666 1991 raft_consensus.cc:2396] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260924 00:22:14.125905 3642 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:14.207391 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 196560us
W20260924 00:22:14.209995 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 191360us
I20260924 00:22:14.211618 2035 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:14.213291 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:14.213361 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.213851 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.214442 2067 raft_consensus.cc:2396] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260924 00:22:14.214931 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:14.215607 1996 raft_consensus.cc:2379] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:14.217300 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:14.217384 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.219676 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.220660 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.221320 1993 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:14.222535 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.223436 2070 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:14.224175 3481 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.230098 3656 consensus_queue.cc:237] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.231433 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:14.231751 3423 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.212445 3653 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:14.233095 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.237967 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.233795 2071 raft_consensus.cc:2396] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260924 00:22:14.234360 2068 raft_consensus.cc:2379] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:14.233953 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:14.235046 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.236271 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.207922 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.261011 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.264426 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:14.265288 3672 log.cc:924] Time spent T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260924 00:22:14.265582 2062 raft_consensus.cc:2396] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260924 00:22:14.273722 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.276245 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.240589 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 166) took 1352 ms. Trace:
I20260924 00:22:14.233574 3653 raft_consensus.cc:686] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:14.275628 3673 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:14.271839 3655 raft_consensus.cc:686] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:14.272394 3612 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.279870 2136 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:14.311020 3653 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:14.301782 3671 log.cc:924] Time spent T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20260924 00:22:14.311851 3529 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:14.311477 3532 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:14.320345 3655 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:14.327790 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 93903us
I20260924 00:22:14.377074 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:14.379148 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 78285us
I20260924 00:22:14.624056 1995 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:14.386530 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.617424 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.625587 1996 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:14.623601 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:14.627578 2614 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:14.316341 3423 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.629637 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.631078 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.631057 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.632087 1996 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:14.633555 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:14.339161 3391 consensus_meta.cc:287] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.001s
I20260924 00:22:14.634687 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:14.636229 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:14.636761 1991 raft_consensus.cc:3032] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:14.637247 1991 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:14.637846 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.327677 3654 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.639039 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.612287 2138 raft_consensus.cc:595] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:14.605963 3505 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:14.495164 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:14.642007 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.571421 3624 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.638125 1991 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.644161 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:14.629086 3673 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:14.650571 1991 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:14.644636 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.388916 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:14.636734 3423 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:14.629086 3520 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260924 00:22:14.641407 3312 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:22:14.648869 3659 consensus_meta.cc:287] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.534s user 0.004s sys 0.006s
I20260924 00:22:14.645751 2138 raft_consensus.cc:603] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:14.631884 3610 consensus_queue.cc:1046] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260924 00:22:14.637799 3532 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260924 00:22:14.648614 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.663861 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.637663 3391 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.667230 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:14.658924 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:14.676034 3610 raft_consensus.cc:3032] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:14.676460 3610 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:14.649470 3505 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.680019 2064 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:14.654917 3582 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:14.683384 3391 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.658082 3624 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.726402 3624 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.663913 3659 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.685439 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:14.729586 3649 consensus_meta.cc:287] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.663s user 0.012s sys 0.000s
I20260924 00:22:14.729764 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.730130 1990 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:14.730086 3649 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.730697 1996 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:14.688967 3505 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.732378 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.734053 3659 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.287271 1987 rpcz_store.cc:276] 0924 00:22:12.888322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:12.891467 (+ 3145us) service_pool.cc:227] Handling call
0924 00:22:13.029253 (+137786us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:13.145727 (+116474us) raft_consensus.cc:1520] Early marking committed up to index 6
0924 00:22:13.274983 (+129256us) log.cc:842] Serialized 14 byte log entry
0924 00:22:13.750237 (+475254us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:13.750291 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:22:13.751269 (+ 978us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:14.022958 (+271689us) spinlock_profiling.cc:243] Waited 6.86 ms on lock 0x7b180014e8a0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d1b94764 00007fb6d1b97b11 00007fb6d1b97ac9 00007fb6d1b97a59 00007fb6d1b97a21 00007fb6d1b96d1d 00007fb6dadf8204 00007fb6dadf8039 00007fb6d1b76dc6
0924 00:22:14.239755 (+216797us) spinlock_profiling.cc:243] Waited 99.6 ms on lock 0x7b5400101154. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:14.239828 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:14.240014 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":175130,"prepare.queue_time_us":84219,"prepare.run_cpu_time_us":8005,"prepare.run_wall_time_us":62993,"raft.queue_time_us":175411,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"spinlock_wait_cycles":244766848,"thread_start_us":221046,"threads_started":1,"wal-append.queue_time_us":297759,"wal-append.run_cpu_time_us":1072,"wal-append.run_wall_time_us":1015068}
I20260924 00:22:14.694813 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.764206 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:14.652585 1989 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:14.689460 2137 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:14.712209 1991 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:14.658923 3423 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.773805 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:14.778352 3040 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:14.782253 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:14.784554 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:14.774164 3423 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.795118 3423 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.771056 3610 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:14.796834 3610 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:14.790511 3040 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:14.798949 3040 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:14.799288 3040 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:14.800746 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:14.808483 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 474776us
I20260924 00:22:14.810745 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:14.811987 3040 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.816423 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.817270 2070 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:14.845795 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:14.849102 3040 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.818336 3481 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:14.845212 3441 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:14.834877 3649 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.850950 3489 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:14.855113 3481 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:14.870103 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 217731us
I20260924 00:22:14.860826 3683 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.901075 2035 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:14.874925 3441 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:14.881724 3489 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:14.909324 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 257520us
I20260924 00:22:14.905079 3655 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:14.881657 3675 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:14.907122 3536 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.493s user 0.031s sys 0.058s
I20260924 00:22:14.885262 3481 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:14.905694 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
W20260924 00:22:14.918857 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 270919us
I20260924 00:22:14.919824 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:14.909763 3441 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:14.910529 3489 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:14.924150 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 66084.9us
I20260924 00:22:14.910597 3655 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:14.925026 3536 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.511s user 0.032s sys 0.059s
W20260924 00:22:14.927112 3687 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260924 00:22:14.932708 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:14.944262 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 90581.2us
I20260924 00:22:14.939612 3655 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:14.909832 3683 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.944145 3392 raft_consensus.cc:3032] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:14.953487 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:14.950184 2110 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:14.921372 3675 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.947062 3610 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.953622 3470 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:14.958906 3685 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:14.955632 3479 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:14.955693 3392 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:14.967336 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.963207 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.956429 2110 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:14.986711 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 56060.4us
I20260924 00:22:14.988727 2110 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:14.988005 2065 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:14.964390 3610 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.989810 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:14.965952 3470 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:14.971104 3675 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:14.970279 3697 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:14.990420 3632 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:14.996647 2110 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:14.986913 3479 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:14.995461 3486 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:22:15.002776 3690 log.cc:924] Time spent T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260924 00:22:15.002907 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:14.973436 3532 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.002111 3692 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260924 00:22:14.998965 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 55103.9us
I20260924 00:22:15.003947 3479 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.000197 3470 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:15.005326 3392 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.017107 3392 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:15.017306 3536 consensus_queue.cc:1159] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3eb374c9744b4c7e92dc749aece177dd" responder_term: 3 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: [2.2-2.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260924 00:22:15.018738 3610 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.007246 3692 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:15.019517 3692 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.021306 3482 consensus_queue.cc:1046] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:15.012116 3683 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.022241 3699 raft_consensus.cc:3032] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:15.022583 3699 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:15.008728 3532 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.025933 3532 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.004659 3632 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:15.035039 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:15.035609 2140 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
W20260924 00:22:15.030301 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 70010us
I20260924 00:22:15.031164 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:15.038262 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:15.038700 2141 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.033898 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:15.038337 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:15.023341 3699 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.051306 2141 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:15.057924 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:15.059077 2144 raft_consensus.cc:2396] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:15.062541 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1069) took 1044 ms. Trace:
I20260924 00:22:15.062930 2064 rpcz_store.cc:276] 0924 00:22:14.017729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:14.017916 (+ 187us) service_pool.cc:227] Handling call
0924 00:22:14.018326 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:14.680496 (+662170us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:15.062294 (+381798us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b54001ecdd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:15.062490 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1521343232}
W20260924 00:22:15.115254 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 74759.3us
W20260924 00:22:15.115638 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 53754.5us
I20260924 00:22:15.116496 2613 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.029629 3702 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:15.041639 3632 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.036273 3705 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:15.151168 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:15.152031 2144 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:15.136250 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:15.174856 3696 log.cc:924] Time spent T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20260924 00:22:15.153692 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:15.166000 3441 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.151440 3705 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:15.192387 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:15.198974 2064 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:15.193081 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.199980 2063 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.201179 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:15.201504 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:15.202345 2064 raft_consensus.cc:2379] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:15.177552 2137 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
W20260924 00:22:15.207691 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 212835us
I20260924 00:22:15.236012 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:15.233443 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:15.238526 2140 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 1.
I20260924 00:22:15.191985 3697 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:15.150122 3702 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.238453 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 1217) took 1025 ms. Trace:
I20260924 00:22:15.253506 2069 rpcz_store.cc:276] 0924 00:22:14.212789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:14.212998 (+ 209us) service_pool.cc:227] Handling call
0924 00:22:15.232125 (+1019127us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b54002b1454. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:15.238386 (+ 6261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1533672832}
I20260924 00:22:15.242916 2064 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:22:15.255097 3040 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260924 00:22:15.260607 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 144214us
W20260924 00:22:15.270062 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 66192.7us
I20260924 00:22:15.271765 2613 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.237982 3546 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260924 00:22:15.278544 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 139816us
I20260924 00:22:15.258113 3040 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:15.280294 3040 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.283658 2063 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:15.276417 2035 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:15.235641 3705 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.245262 3697 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.290860 2035 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:15.256920 3521 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20260924 00:22:15.241816 3470 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.269258 3520 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260924 00:22:15.308251 3709 log.cc:924] Time spent T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260924 00:22:15.241657 3481 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.313200 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:15.278872 3546 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.313803 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:15.330292 3632 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.333393 2142 raft_consensus.cc:1226] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:15.337519 3702 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.341879 3692 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.344545 3692 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.301172 3697 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:15.347731 1989 consensus_meta.cc:287] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.574s user 0.005s sys 0.006s
I20260924 00:22:15.348246 1989 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:15.332851 3632 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.302264 3423 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:15.362896 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.310142 3612 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:15.366719 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.350256 3546 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.304859 3441 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.324340 2136 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:15.349754 3454 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:15.399632 3612 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.401427 3423 raft_consensus.cc:686] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
W20260924 00:22:15.420251 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 158882us
W20260924 00:22:15.422319 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 186416us
I20260924 00:22:15.417883 3612 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.421023 3521 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:15.451107 3040 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.456914 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.458355 3699 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.465232 3423 consensus_queue.cc:237] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.469509 2138 raft_consensus.cc:1261] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:15.473554 2646 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 00:22:15.473901 3470 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.428246 3489 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.482360 2643 raft_consensus.cc:1226] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:15.483147 3489 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.486104 3040 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.447739 3683 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:15.489537 3040 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260924 00:22:15.493824 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 205786us
I20260924 00:22:15.494896 2109 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:15.494606 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:15.496335 2109 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.498092 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:15.448647 3521 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.506108 2109 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:15.513554 3470 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:15.501863 3702 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:15.515827 3702 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:15.516286 3702 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:15.518009 3720 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:15.518415 3720 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:15.518771 3720 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:15.521272 3529 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:15.521725 3529 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.522090 3529 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.501767 3546 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.524682 3546 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.533025 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33238 (request call id 228) took 6016 ms. Trace:
I20260924 00:22:15.533465 1893 rpcz_store.cc:276] 0924 00:22:09.516902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:09.517074 (+ 172us) service_pool.cc:227] Handling call
0924 00:22:15.532959 (+6015885us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:13.557034 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:13.628277 (+ 71243us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:13.628311 (+ 34us) write_op.cc:435] Acquired schema lock
0924 00:22:13.628326 (+ 15us) tablet.cc:660] Decoding operations
0924 00:22:13.628983 (+ 657us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:13.629067 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:13.629095 (+ 28us) tablet.cc:683] Acquiring locks for 5 operations
0924 00:22:13.629656 (+ 561us) tablet.cc:699] Row locks acquired
0924 00:22:13.629673 (+ 17us) write_op.cc:260] PREPARE: finished
0924 00:22:13.629814 (+ 141us) write_op.cc:270] Start()
0924 00:22:13.629917 (+ 103us) write_op.cc:276] Timestamp: P: 1790209333629771 usec, L: 0
0924 00:22:13.629936 (+ 19us) op_driver.cc:331] REPLICATION: starting
0924 00:22:13.630522 (+ 586us) log.cc:842] Serialized 2064 byte log entry
0924 00:22:13.681936 (+ 51414us) op_driver.cc:444] REPLICATION: finished
0924 00:22:13.721975 (+ 40039us) write_op.cc:301] APPLY: starting
0924 00:22:13.722059 (+ 84us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:13.722212 (+ 153us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:13.722336 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:13.723052 (+ 716us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:13.723231 (+ 179us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0924 00:22:13.723273 (+ 42us) write_op.cc:312] APPLY: finished
0924 00:22:13.723488 (+ 215us) log.cc:842] Serialized 47 byte log entry
0924 00:22:13.723644 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:13.723747 (+ 103us) write_op.cc:454] Released schema lock
0924 00:22:13.723800 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:13.724034 (+ 234us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:13.724056 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23944,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":39767,"apply.run_cpu_time_us":2326,"apply.run_wall_time_us":2388,"num_ops":5,"prepare.queue_time_us":749621,"prepare.run_cpu_time_us":9521,"prepare.run_wall_time_us":111313,"raft.queue_time_us":6919,"raft.run_cpu_time_us":1330,"raft.run_wall_time_us":7553,"replication_time_us":51915,"thread_start_us":688526,"threads_started":4,"wal-append.queue_time_us":39486,"wal-append.run_cpu_time_us":9209,"wal-append.run_wall_time_us":1062099}]]}
I20260924 00:22:15.538647 3529 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.541311 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:15.542485 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:15.543251 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 256200us
I20260924 00:22:15.547593 2109 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.503152 3692 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260924 00:22:15.543274 3655 consensus_meta.cc:287] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.585s user 0.011s sys 0.000s
I20260924 00:22:15.559132 2109 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.495661 3683 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.561492 1996 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:15.561197 3685 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:15.547971 3684 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:15.548141 3520 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 00:22:15.566104 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 68221.2us
I20260924 00:22:15.564188 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.567045 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.556557 3692 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:15.569023 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:15.528977 3720 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.569101 2109 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.571349 3521 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.572014 3441 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:15.566512 3520 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:15.572918 3520 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:15.566578 3724 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:15.574337 3724 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.574725 3724 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.575246 3529 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.575991 1988 raft_consensus.cc:1226] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:15.575927 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:15.577044 3482 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:15.577499 3482 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:15.577616 3685 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:15.577917 3482 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.566946 3642 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260924 00:22:15.583906 3692 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.585660 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:15.585887 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.586551 1987 raft_consensus.cc:2396] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:15.590627 3683 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.568495 3698 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:15.561925 3697 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.591885 3482 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.593794 3697 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.560288 3655 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.596521 3655 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.602289 3705 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.603343 1992 raft_consensus.cc:1226] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:15.604204 3705 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.607766 1992 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:15.609174 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.609411 3632 consensus_queue.cc:1046] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:15.610051 3482 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.608878 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:15.610526 3479 raft_consensus.cc:3032] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:15.611800 3479 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:15.612098 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.631219 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.633414 3040 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:15.612809 3479 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.635200 1991 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:15.635658 3479 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.637395 3532 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:15.637789 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:15.573832 3481 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.639365 3481 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:15.639727 3481 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:15.640060 3481 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:15.640676 3702 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:15.643323 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 362498us
I20260924 00:22:15.644124 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:15.630734 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 61713.6us
I20260924 00:22:15.645576 2109 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.648849 3040 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:15.649258 3040 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:15.649618 3040 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:15.650161 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.651803 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 127) took 2058 ms. Trace:
I20260924 00:22:15.653185 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.653162 1892 catalog_manager.cc:5495] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:15.654204 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:15.655134 1892 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:15.656814 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.630509 3698 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260924 00:22:15.660125 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 51711.9us
I20260924 00:22:15.630162 3482 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:15.633006 3536 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:15.663319 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:15.664438 3521 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:15.663322 2109 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.666090 3521 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:15.667204 3482 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:15.667354 2109 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.667656 3482 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:15.667474 3692 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.668182 3482 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:15.671031 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.675139 3040 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.653062 3441 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:15.677062 3521 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:15.677239 3520 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.677973 3692 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.679317 3520 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.681463 3642 consensus_queue.cc:1046] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:15.682308 3692 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:15.682520 3642 raft_consensus.cc:3032] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:15.682977 3642 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:15.657342 3632 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:15.683471 3720 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.683836 3642 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.669713 3536 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.686115 3536 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.663420 3702 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.689350 2109 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.691870 2109 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.673401 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.694689 2063 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.695152 3040 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.695556 2109 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.656620 1892 catalog_manager.cc:5495] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:15.696110 2141 raft_consensus.cc:1226] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:15.697459 1892 catalog_manager.cc:5495] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:15.685060 3642 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.702229 3481 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.704185 3481 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:15.704946 3632 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:15.705351 3632 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.705720 3632 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.707198 3643 raft_consensus.cc:3032] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:15.707571 3643 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:15.708328 3643 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.682231 3423 consensus_queue.cc:1046] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260924 00:22:15.709246 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.709980 3479 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:15.710042 3423 raft_consensus.cc:3032] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:15.710175 3643 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.710506 3479 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.662369 3698 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:15.711133 3479 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:15.711211 3698 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.712278 3632 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.710563 3423 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:15.713812 3423 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.715035 3423 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.669839 3697 consensus_queue.cc:1046] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260924 00:22:15.719792 3724 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.722023 3724 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.722311 3632 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.652169 1891 rpcz_store.cc:276] 0924 00:22:13.600100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:13.622025 (+ 21925us) service_pool.cc:227] Handling call
0924 00:22:15.651749 (+2029724us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:14.957858 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:14.973785 (+ 15927us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:14.973826 (+ 41us) write_op.cc:435] Acquired schema lock
0924 00:22:14.973845 (+ 19us) tablet.cc:660] Decoding operations
0924 00:22:14.974549 (+ 704us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:14.974612 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:14.974641 (+ 29us) tablet.cc:683] Acquiring locks for 5 operations
0924 00:22:14.975243 (+ 602us) tablet.cc:699] Row locks acquired
0924 00:22:14.975269 (+ 26us) write_op.cc:260] PREPARE: finished
0924 00:22:14.975412 (+ 143us) write_op.cc:270] Start()
0924 00:22:14.975524 (+ 112us) write_op.cc:276] Timestamp: P: 1790209334975365 usec, L: 0
0924 00:22:14.975546 (+ 22us) op_driver.cc:331] REPLICATION: starting
0924 00:22:14.976204 (+ 658us) log.cc:842] Serialized 2039 byte log entry
0924 00:22:15.514228 (+538024us) op_driver.cc:444] REPLICATION: finished
0924 00:22:15.634949 (+120721us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b1800199de0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d1b94638 00007fb6d1b97b11 00007fb6d1b97ac9 00007fb6d1b97a59 00007fb6d1b97a21 00007fb6d1b96d1d 00007fb6dadf8204 00007fb6dadf8039 00007fb6d1b76dc6
0924 00:22:15.635105 (+ 156us) write_op.cc:301] APPLY: starting
0924 00:22:15.635182 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:15.635332 (+ 150us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:15.635354 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:15.636005 (+ 651us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:15.636186 (+ 181us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0924 00:22:15.636241 (+ 55us) write_op.cc:312] APPLY: finished
0924 00:22:15.636491 (+ 250us) log.cc:842] Serialized 47 byte log entry
0924 00:22:15.636667 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:15.636765 (+ 98us) write_op.cc:454] Released schema lock
0924 00:22:15.636811 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:15.637058 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:15.637084 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":995843,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":67210,"apply.run_cpu_time_us":2306,"apply.run_wall_time_us":2306,"num_ops":5,"prepare.queue_time_us":175613,"prepare.run_cpu_time_us":11525,"prepare.run_wall_time_us":66184,"raft.queue_time_us":204789,"raft.run_cpu_time_us":13995,"raft.run_wall_time_us":56142,"replication_time_us":538571,"spinlock_wait_cycles":111738624,"thread_start_us":382150,"threads_started":4,"wal-append.queue_time_us":177701}]]}
I20260924 00:22:15.723277 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.723547 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.690867 3612 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.727229 3612 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.728011 3697 raft_consensus.cc:3032] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:15.728422 3697 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:15.729302 3697 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:15.730435 3697 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.733523 3627 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:15.733994 3627 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:15.734860 3627 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.735236 2109 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.737516 3521 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:15.737654 2109 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.738162 3521 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:15.738449 3685 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:15.734185 3479 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.740272 3482 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.740823 3479 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:15.738822 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.743858 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.744671 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:15.698398 3692 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.745533 3423 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:15.745918 3423 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.745951 3627 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:15.746851 3423 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:15.749084 3479 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:15.749212 3520 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:15.742529 3482 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.754076 3479 raft_consensus.cc:686] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:15.756559 3479 consensus_queue.cc:237] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.757861 1992 raft_consensus.cc:1226] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:15.736646 3536 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.766242 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.761394 1991 raft_consensus.cc:1226] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:15.762037 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:15.762629 2063 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:15.768927 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.772156 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.772903 2068 raft_consensus.cc:2379] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:15.772927 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.767956 2069 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:15.774981 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.765478 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:15.779805 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.780956 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:15.784123 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 129987us
I20260924 00:22:15.783660 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:15.783660 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.785442 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:15.786240 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:15.786584 2066 raft_consensus.cc:2379] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:15.787331 2068 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.789352 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:15.790863 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.793886 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:15.797739 2068 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:15.798979 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:15.797269 2613 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:15.801529 3040 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:15.801915 3040 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:15.802306 3040 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.808161 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:15.809597 3649 consensus_queue.cc:1046] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:15.810665 3649 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:15.811003 3649 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:15.811832 3649 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.812978 3649 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:15.789871 3582 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:15.791951 3698 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.783512 3520 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.822383 2142 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:15.791412 3521 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:15.827745 3040 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.830569 3040 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.799413 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:15.834342 3698 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:15.834587 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:15.835253 2137 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:15.989914 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:15.837245 3521 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:15.806833 3685 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.185s
I20260924 00:22:15.796279 3481 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:15.823731 3593 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:15.790370 3683 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260924 00:22:15.792060 3482 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:15.777791 3706 consensus_queue.cc:1046] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260924 00:22:15.805377 3726 consensus_queue.cc:1035] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.190s
I20260924 00:22:15.837210 3642 consensus_queue.cc:1046] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.160s
W20260924 00:22:15.997414 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 189326us
I20260924 00:22:15.791604 3699 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:15.797956 3700 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:15.821395 3520 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:16.034475 1942 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20260924 00:22:15.991833 3040 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:16.049772 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.051530 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:16.052186 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.090953 2068 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:16.073773 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.093408 2144 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:16.079662 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.086120 3675 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:16.055437 3536 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:16.094576 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.073082 2138 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:16.090397 2142 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:16.061590 3684 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:16.101662 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.102241 3040 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:16.095274 2143 raft_consensus.cc:2379] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:16.074991 3482 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:16.113497 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:16.103112 2068 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:16.088289 2136 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:16.122525 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.088829 3700 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:16.121352 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.130991 2070 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 5
I20260924 00:22:16.133397 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.133975 2065 raft_consensus.cc:2362] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3. Re-sending same reply.
I20260924 00:22:16.126827 2068 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 6
I20260924 00:22:16.120493 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:16.246868 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 126450us
I20260924 00:22:16.248523 2036 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:16.119038 3040 consensus_queue.cc:1046] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:16.249972 3040 raft_consensus.cc:3032] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:16.250365 3040 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:16.251148 3040 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.252432 3040 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:16.236370 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:16.253682 2062 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:16.078167 3481 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.261327 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:16.266098 2068 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:16.267516 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.269029 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.099766 3699 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:16.244073 2144 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:16.245460 2069 raft_consensus.cc:1226] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260924 00:22:16.118682 3482 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:16.272187 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.124174 3702 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:16.110015 3532 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:16.279632 1991 raft_consensus.cc:1226] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.281885 1992 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:16.284437 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:16.286134 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 970b194b81a34f3db37adc13d58b0a01, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.288604 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:16.292260 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 970b194b81a34f3db37adc13d58b0a01, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.292438 2143 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:16.237082 3700 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:16.295032 2144 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:16.298062 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.301051 2062 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:16.298482 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.270901 3391 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:16.301868 2143 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:16.303268 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.289808 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.303892 2143 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.304777 3040 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:16.305647 3040 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.306253 1960 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:16.307591 1960 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:16.307986 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:16.308591 2069 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:16.308715 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:16.309660 3040 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:16.311918 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.314229 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.315068 1991 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:16.309827 3391 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:16.316520 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.318540 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.321187 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.326118 2143 raft_consensus.cc:1226] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.304159 2068 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:16.327857 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.328763 1987 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:16.321775 1992 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:16.330855 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:16.124282 3684 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.293004 3733 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:16.334327 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.324644 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.348212 2065 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.340631 2070 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:16.353602 2070 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.356594 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.362012 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.364763 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.366641 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.371831 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.374045 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.374337 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:16.316778 3739 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
I20260924 00:22:16.379357 2144 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.379941 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.387580 2140 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:16.301931 3683 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:16.382143 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.383801 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.420483 3481 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:16.422838 2138 raft_consensus.cc:1226] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.428627 3684 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:16.330062 3391 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:16.433054 2062 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:16.407325 2141 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:16.434973 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.384521 2070 raft_consensus.cc:1226] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.310307 3699 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:16.313344 3698 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:16.387704 2139 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:16.381268 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.438987 2143 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.439658 1992 raft_consensus.cc:2354] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:16.393015 2068 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:16.290427 3532 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.394753 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.316939 3720 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:16.397339 3733 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.398046 1989 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:16.406618 2140 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:16.408442 2065 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
W20260924 00:22:16.407282 3692 consensus_meta.cc:287] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.661s user 0.008s sys 0.001s
I20260924 00:22:16.441355 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.441908 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.450027 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.450963 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.441248 2069 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:16.443799 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:16.407649 3423 consensus_meta.cc:287] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.660s user 0.008s sys 0.001s
I20260924 00:22:16.446493 2109 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.462075 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.463806 1994 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:16.413933 3683 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:16.465644 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:16.467136 1991 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:16.471210 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:16.454931 2643 consensus_meta.cc:287] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Time spent flushing consensus metadata: real 0.656s user 0.007s sys 0.002s
I20260924 00:22:16.452591 3521 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:16.458071 3692 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:16.467984 3684 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:16.456756 3593 raft_consensus.cc:686] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:16.483924 2140 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.471971 3683 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:16.486299 2140 raft_consensus.cc:1226] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:16.487730 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.488584 2142 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.491626 2109 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.475368 2643 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:16.465592 3423 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.484205 3593 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.440287 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:16.649397 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 157715us
I20260924 00:22:16.654142 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:16.654614 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:16.655083 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:16.636595 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.657003 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.657820 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:16.659152 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.665961 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.627205 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:16.639760 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 391133us
I20260924 00:22:16.682008 1993 raft_consensus.cc:2362] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3. Re-sending same reply.
I20260924 00:22:16.683171 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:16.638895 2071 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.009s sys 0.001s
I20260924 00:22:16.674434 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.534124 3582 raft_consensus.cc:3032] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:16.686086 3582 raft_consensus.cc:729] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:16.686122 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.687753 3582 consensus_queue.cc:260] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.669761 3529 consensus_queue.cc:1046] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:16.692986 3655 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:16.694103 3582 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:16.671686 3423 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:16.670466 2109 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.673517 3481 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:16.701407 3632 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:16.701249 3481 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.703197 3632 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:16.703896 3481 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:16.669781 3532 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:16.669818 2067 raft_consensus.cc:2379] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:16.579038 3700 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.703814 3632 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.706159 3532 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:16.627781 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.639449 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.677796 3478 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260924 00:22:16.483948 3733 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:16.612208 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 47c6ea1617904bf9bf2b2d904d0e2ae5, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.680816 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.630259 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:16.685660 2071 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s user 0.009s sys 0.001s
I20260924 00:22:16.683303 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.688727 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:16.679925 2642 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:16.709976 1989 raft_consensus.cc:2354] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:16.708767 2069 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.709275 2109 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.667773 3482 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:16.716552 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 47c6ea1617904bf9bf2b2d904d0e2ae5, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.492775 3750 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:16.708663 3529 raft_consensus.cc:3032] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:16.733841 3529 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:16.492944 3738 consensus_queue.cc:1046] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.242s
I20260924 00:22:16.711179 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.735881 3738 raft_consensus.cc:3032] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:16.736374 3738 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:16.736892 1993 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.738538 3529 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.522405 3753 consensus_queue.cc:1046] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.222s
I20260924 00:22:16.740857 3753 raft_consensus.cc:3032] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:16.741300 3753 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:16.742656 3753 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.744076 1987 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:16.744377 3753 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:16.745615 1987 raft_consensus.cc:1226] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:16.747005 1987 raft_consensus.cc:1226] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:16.748014 3738 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.749650 3738 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:16.752154 3529 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:16.703619 3683 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.715444 3700 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:16.723312 1988 raft_consensus.cc:2362] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4. Re-sending same reply.
I20260924 00:22:16.758793 3749 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:16.761421 2644 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.759778 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.731027 3632 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:16.760385 3423 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:16.774451 3423 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.777423 3520 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:16.776844 3423 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:16.783577 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:22:16.795004 3627 consensus_meta.cc:287] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 1.048s user 0.008s sys 0.012s
I20260924 00:22:16.794513 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:16.804198 2069 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.804684 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 1248) took 1015 ms. Trace:
I20260924 00:22:16.804930 2065 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:16.798513 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:16.810123 2067 rpcz_store.cc:276] 0924 00:22:15.788914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:15.789062 (+ 148us) service_pool.cc:227] Handling call
0924 00:22:16.804416 (+1015354us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b5400306454. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:16.804607 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2022166784}
I20260924 00:22:16.815915 3521 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:16.816372 3521 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:22:16.820495 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 112583us
I20260924 00:22:16.824898 1996 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:16.828877 1994 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260924 00:22:16.831660 2145 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
W20260924 00:22:16.832825 2137 consensus_meta.cc:287] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.997s user 0.018s sys 0.000s
I20260924 00:22:16.830675 3521 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.835536 3391 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.797056 1990 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:16.846026 3593 consensus_queue.cc:1046] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260924 00:22:16.841831 2109 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:16.845579 1991 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:16.845381 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:16.867102 3521 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:16.884779 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1234) took 1117 ms. Trace:
I20260924 00:22:16.885711 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:16.868844 3685 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:16.844147 2137 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:16.871176 3470 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:16.871987 3693 raft_consensus.cc:3032] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:16.872689 3692 consensus_queue.cc:1046] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:16.866812 2640 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:16.878533 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:16.878925 3683 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:16.847209 3742 log.cc:924] Time spent T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.589s user 0.002s sys 0.001s
I20260924 00:22:16.856523 3482 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:16.867110 2109 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:16.881394 3391 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:16.869052 3724 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:16.885802 2071 rpcz_store.cc:276] 0924 00:22:15.767199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:15.769261 (+ 2062us) service_pool.cc:227] Handling call
0924 00:22:15.769542 (+ 281us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 00:22:15.781772 (+ 12230us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:15.781800 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 00:22:15.801012 (+ 19212us) log.cc:842] Serialized 52 byte log entry
0924 00:22:15.815980 (+ 14968us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b4800210258. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d99637f1 00007fb6d995a8e7 00007fb6d995a77a 00007fb6d9a4e95f 00007fb6d9a4e627 00007fb6d9a62b2c 00007fb6d9a62ad8 00007fb6d9a62a58 00007fb6d9a62a13 00007fb6d9a61cbc 00007fb6d89e3be6
0924 00:22:16.726740 (+910760us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:16.789469 (+ 62729us) log.cc:842] Serialized 14 byte log entry
0924 00:22:16.883081 (+ 93612us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b4800210258. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d99637f1 00007fb6d995a8e7 00007fb6d995a77a 00007fb6d9a4e95f 00007fb6d9a4e627 00007fb6d9a62b2c 00007fb6d9a62ad8 00007fb6d9a62a58 00007fb6d9a62a13 00007fb6d9a61cbc 00007fb6d89e3be6
0924 00:22:16.884001 (+ 920us) log.cc:842] Serialized 14 byte log entry
0924 00:22:16.884252 (+ 251us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:16.884422 (+ 170us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:16.884494 (+ 72us) raft_consensus.cc:1671] finished
0924 00:22:16.884515 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:16.884701 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7758,"prepare.queue_time_us":86241,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":39197,"raft.run_cpu_time_us":5732,"raft.run_wall_time_us":68826,"spinlock_wait_cycles":47913600,"thread_start_us":722937,"threads_started":2,"wal-append.queue_time_us":615541}
I20260924 00:22:16.888706 3697 consensus_queue.cc:1046] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:16.891002 2069 raft_consensus.cc:1226] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:16.788326 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:16.895330 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:16.905516 3692 raft_consensus.cc:3032] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:16.896245 3693 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:16.909106 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1232) took 1146 ms. Trace:
I20260924 00:22:16.915277 2064 rpcz_store.cc:276] 0924 00:22:15.762686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:15.762831 (+ 145us) service_pool.cc:227] Handling call
0924 00:22:15.763114 (+ 283us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:15.763453 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:15.763493 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:15.763871 (+ 378us) log.cc:842] Serialized 26 byte log entry
0924 00:22:16.057628 (+293757us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:16.057683 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:16.057770 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:16.846267 (+788497us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:16.904127 (+ 57860us) raft_consensus.cc:1671] finished
0924 00:22:16.904151 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:16.908959 (+ 4808us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1242,"raft.queue_time_us":2595,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5452,"spinlock_wait_cycles":293132672,"thread_start_us":291844,"threads_started":1,"wal-append.queue_time_us":326989}
I20260924 00:22:16.911940 3479 raft_consensus.cc:3032] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:16.917866 3479 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:16.918988 3479 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.920680 3479 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:16.912791 1994 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:16.912925 3700 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:16.922662 3700 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:16.923071 3700 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:16.913939 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60184 (request call id 1530) took 1091 ms. Trace:
I20260924 00:22:16.925832 2137 rpcz_store.cc:276] 0924 00:22:15.822737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:15.829455 (+ 6718us) service_pool.cc:227] Handling call
0924 00:22:16.906587 (+1077132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:16.928584 3697 raft_consensus.cc:3032] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:16.929044 3697 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:16.928124 1891 catalog_manager.cc:5495] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:16.930159 3697 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.931845 3697 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:16.933667 3692 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:16.935245 3692 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.914764 3693 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:16.939621 3693 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:16.942071 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 75197.8us
I20260924 00:22:16.943423 3521 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:16.944065 3521 raft_consensus.cc:686] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:16.944120 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:16.946027 2109 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:16.946650 3470 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:16.945446 3521 consensus_queue.cc:237] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.947391 3632 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:16.947809 2109 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:16.948239 3724 raft_consensus.cc:686] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:16.949420 3724 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:16.957513 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33246 (request call id 215) took 4542 ms. Trace:
I20260924 00:22:16.957592 3692 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:16.932945 1891 catalog_manager.cc:5495] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:16.958762 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:16.970120 1892 rpcz_store.cc:276] 0924 00:22:12.415297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:12.415454 (+ 157us) service_pool.cc:227] Handling call
0924 00:22:16.957285 (+4541831us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:16.299311 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:16.397152 (+ 97841us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:16.414221 (+ 17069us) write_op.cc:435] Acquired schema lock
0924 00:22:16.418348 (+ 4127us) tablet.cc:660] Decoding operations
0924 00:22:16.419364 (+ 1016us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:16.419464 (+ 100us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:16.419519 (+ 55us) tablet.cc:683] Acquiring locks for 4 operations
0924 00:22:16.420063 (+ 544us) tablet.cc:699] Row locks acquired
0924 00:22:16.420093 (+ 30us) write_op.cc:260] PREPARE: finished
0924 00:22:16.420305 (+ 212us) write_op.cc:270] Start()
0924 00:22:16.420507 (+ 202us) write_op.cc:276] Timestamp: P: 1790209336420247 usec, L: 0
0924 00:22:16.420540 (+ 33us) op_driver.cc:331] REPLICATION: starting
0924 00:22:16.468782 (+ 48242us) log.cc:842] Serialized 1682 byte log entry
0924 00:22:16.585377 (+116595us) op_driver.cc:444] REPLICATION: finished
0924 00:22:16.586687 (+ 1310us) write_op.cc:301] APPLY: starting
0924 00:22:16.586779 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:16.586942 (+ 163us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:16.587075 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:16.587590 (+ 515us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:16.587798 (+ 208us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0924 00:22:16.587882 (+ 84us) write_op.cc:312] APPLY: finished
0924 00:22:16.588121 (+ 239us) log.cc:842] Serialized 41 byte log entry
0924 00:22:16.725283 (+137162us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:16.725593 (+ 310us) write_op.cc:454] Released schema lock
0924 00:22:16.728853 (+ 3260us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:16.740242 (+ 11389us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:16.740273 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4418,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":1077,"apply.run_cpu_time_us":3465,"apply.run_wall_time_us":154058,"num_ops":4,"prepare.queue_time_us":464343,"prepare.run_cpu_time_us":4625,"prepare.run_wall_time_us":209940,"replication_time_us":164728,"thread_start_us":304023,"threads_started":2}]]}
W20260924 00:22:16.979975 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 102636us
I20260924 00:22:16.988093 3391 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:16.993757 3593 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:16.993752 3482 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:16.988257 3738 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:16.986429 3685 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:16.995899 3482 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:16.998988 2036 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:16.994647 3593 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.014977 2036 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.014729 3482 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.000041 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:16.995191 3642 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:17.016559 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.016925 3683 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:17.016989 3642 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.017781 3683 raft_consensus.cc:2741] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260924 00:22:17.006953 3738 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.004704 3391 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.021741 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.010282 2145 raft_consensus.cc:1226] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:17.010520 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.023554 2142 raft_consensus.cc:2379] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:17.023765 3391 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:17.010850 3733 consensus_queue.cc:1046] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:17.024834 3733 raft_consensus.cc:3032] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:17.025192 3733 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.025331 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:17.012923 3040 consensus_queue.cc:1046] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:17.026193 2640 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:17.016371 2036 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.028272 2036 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.029072 3040 raft_consensus.cc:3032] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:17.029498 3040 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.029832 2036 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.009397 3705 consensus_queue.cc:1046] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:17.030382 3040 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.030737 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:17.031078 3705 raft_consensus.cc:3032] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:17.031512 3705 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.031638 3040 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:17.031769 3738 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.032251 3683 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:17.032877 3655 raft_consensus.cc:2741] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260924 00:22:17.033017 3683 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:17.033609 3683 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:17.033618 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.034214 3683 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:17.035009 3642 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.035732 3593 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.013171 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:17.037346 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.037307 3705 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.021764 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.038338 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:16.987323 3700 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:17.040035 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 92355.2us
I20260924 00:22:17.033681 3655 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:17.056232 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.056548 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.057468 2613 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.056222 3702 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:17.058254 3702 raft_consensus.cc:686] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.051946 3655 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:17.058444 3697 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:17.058796 3655 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:17.059342 3697 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:17.059372 3702 consensus_queue.cc:237] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.056984 2136 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:17.026194 3733 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.061890 3733 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:17.065728 2613 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.069545 3705 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:17.072672 3693 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:17.080062 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:17.080818 2109 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:17.080976 3481 consensus_queue.cc:1046] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:17.081810 3529 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:17.082403 3481 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:17.082933 3481 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.084043 3481 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.082422 3529 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:17.092854 3040 raft_consensus.cc:3032] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:17.096174 3040 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.101540 2035 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:17.105386 3391 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:17.105890 3391 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:17.106309 3391 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:17.106611 3612 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:17.111129 3683 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.112530 3700 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.086393 3529 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.121778 2035 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:17.123387 3423 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:17.123239 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:17.124006 3423 raft_consensus.cc:686] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:17.108806 3612 consensus_queue.cc:1046] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20260924 00:22:17.124706 3675 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:17.125237 3675 raft_consensus.cc:686] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:17.124945 3423 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.126202 3675 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:17.132987 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 45988.5us
I20260924 00:22:17.124972 3612 raft_consensus.cc:3032] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:17.137632 1987 raft_consensus.cc:3032] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:17.139753 1987 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.140784 1987 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.141963 1987 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:17.138028 3612 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:17.146265 3612 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.116940 3391 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.148052 3040 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.149156 2110 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:17.150596 3756 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:17.150792 3479 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:17.150758 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "771618125e684526a907a103e7694ae8"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:17.151432 3479 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:17.151279 3756 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.151715 1995 raft_consensus.cc:595] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:17.151858 3479 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:17.152107 1995 raft_consensus.cc:603] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260924 00:22:17.150030 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 57295.7us
I20260924 00:22:17.156622 2036 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.157950 3546 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:17.158361 2036 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.158710 3700 consensus_queue.cc:1046] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:17.160137 3756 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:17.160336 3482 raft_consensus.cc:3032] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:17.160775 3482 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.161935 3482 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.160797 3756 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:17.163291 1995 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:17.120958 3655 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.172379 1987 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260924 00:22:17.165359 3684 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260924 00:22:17.173938 3684 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:17.173720 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:17.174322 3684 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:17.174552 2640 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:17.174698 3593 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:17.175298 3684 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.176513 3684 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:17.177891 3482 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:17.174197 2036 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.182085 2035 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:17.183815 3756 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:17.184480 3756 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:17.184825 3756 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:17.185187 3756 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:17.185273 3612 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:17.185895 3391 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.183012 2036 leader_election.cc:398] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:17.188199 2036 leader_election.cc:401] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:17.182905 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:17.190169 3700 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:17.190755 3700 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.191237 3040 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:17.192250 3700 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.201834 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:17.202844 3733 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:17.202807 3481 consensus_queue.cc:1046] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:17.203406 3733 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:17.203627 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.203809 3733 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:17.204077 3481 raft_consensus.cc:3032] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:17.204490 3481 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.200621 3655 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:17.205432 3481 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.206346 3705 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:17.206732 3481 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:17.206920 3705 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.208071 3705 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.207136 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:17.211603 3733 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.214201 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:17.214862 2640 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:17.215720 3733 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.218256 3423 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:17.226249 2137 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260924 00:22:17.229224 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 46358.3us
I20260924 00:22:17.227144 3481 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:17.244086 3683 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.252316 2640 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:17.263456 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:17.264773 3683 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:17.265765 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.266657 2145 raft_consensus.cc:2379] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:17.252344 3479 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.272347 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.273830 3479 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:17.273700 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.274612 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:17.274637 3692 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:17.286254 3692 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.288146 3692 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:17.282573 3612 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:17.292678 2140 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
W20260924 00:22:17.283674 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 76602us
I20260924 00:22:17.293756 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.294348 3612 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:17.294857 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 59381.7us
I20260924 00:22:17.298815 3391 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:17.265806 3683 raft_consensus.cc:686] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:17.282814 3756 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.308364 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.309195 1991 raft_consensus.cc:2379] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:17.324733 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.328706 2036 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.332511 3040 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:17.313751 3391 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.339072 3481 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:17.340155 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.341900 2145 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:17.342269 3040 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.344053 3040 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.353348 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.397212 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:17.339836 3675 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:17.401798 2139 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:17.403167 3700 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:17.404927 3655 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:17.407353 3612 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.409529 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:17.410503 2642 raft_consensus.cc:2379] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:17.413769 2035 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:17.413794 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 92376.1us
I20260924 00:22:17.415998 2036 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.400091 3481 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.417809 3481 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.418051 2036 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.418489 3423 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:17.418908 3546 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:17.419898 3546 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:17.420398 3649 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:17.358402 3683 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.424896 2064 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260924 00:22:17.428630 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.419059 3423 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:17.430061 3423 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.431349 2139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:17.431912 2139 raft_consensus.cc:595] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:17.432236 2139 raft_consensus.cc:603] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:17.436990 1990 raft_consensus.cc:1261] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:17.437608 3391 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:17.440119 2109 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:17.451910 3692 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:17.445891 2036 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.755s (SENT) [suppressed 29 similar messages]
I20260924 00:22:17.444406 3697 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260924 00:22:17.450400 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:17.448305 3529 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260924 00:22:17.451251 3482 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:17.462654 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.465487 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.458279 3521 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:17.468446 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:17.458489 3756 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:17.481949 2071 raft_consensus.cc:2379] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:17.484267 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.484767 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.484890 2067 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:17.484886 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.486397 2069 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:17.486838 2062 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:17.488550 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.492090 2139 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:17.488672 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.493491 2064 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:17.485671 2070 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:17.494911 3040 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:17.464123 3423 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:17.509997 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 47420.7us
I20260924 00:22:17.511679 3040 raft_consensus.cc:686] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.486850 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.482534 3482 raft_consensus.cc:686] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.518680 3040 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.517863 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.522886 2068 raft_consensus.cc:2379] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 2: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:17.480127 3521 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.526638 2035 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:17.532294 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:17.533363 2065 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:17.535455 3733 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:17.536058 3733 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.535889 3482 consensus_queue.cc:237] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.537083 3733 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.534595 2035 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:17.522442 3756 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:17.535202 3683 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:17.542740 3683 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:17.543162 3683 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:17.521090 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba683df200644d888896fb84dbc748e8"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:17.558089 2065 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:17.518169 3481 consensus_queue.cc:1046] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260924 00:22:17.515875 3423 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.576349 2109 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:17.601703 3392 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:17.561939 2136 raft_consensus.cc:595] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:17.607617 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 65841.3us
I20260924 00:22:17.614048 2068 raft_consensus.cc:1261] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:17.549715 3756 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:17.621735 2036 leader_election.cc:334] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.764s (SENT)
I20260924 00:22:17.622341 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.628046 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:17.613592 2136 raft_consensus.cc:603] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:17.624439 3771 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:17.645048 3702 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:17.645471 3529 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:17.658514 3700 raft_consensus.cc:3032] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:17.659190 3700 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.660101 3700 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.658627 3756 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.644068 3481 consensus_queue.cc:1046] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
W20260924 00:22:17.728946 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 101014us
I20260924 00:22:17.730798 1991 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:17.732049 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:17.733592 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 111901us
I20260924 00:22:17.737440 1996 raft_consensus.cc:2379] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:17.736232 3700 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:17.653461 3529 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:17.646090 3392 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:17.754997 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 178807us
I20260924 00:22:17.739804 3521 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:17.761621 3521 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260924 00:22:17.764168 2036 leader_election.cc:334] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.776s (SENT)
I20260924 00:22:17.741232 3756 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:17.741308 3724 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:17.724668 3771 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.748844 3529 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:17.799350 3392 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:17.800884 2644 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
W20260924 00:22:17.801462 2036 leader_election.cc:334] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.886s (SENT)
I20260924 00:22:17.802627 1990 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:17.803177 3724 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:17.774989 3790 log.cc:924] Time spent T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260924 00:22:17.812059 3683 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.751603 3423 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.755981 3546 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:17.818092 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:17.848086 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.849077 1991 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:17.842888 2036 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.812883 3705 raft_consensus.cc:3032] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:17.854910 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:17.844550 3692 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:17.873274 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.873772 2142 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:17.874666 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.875174 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.875387 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.889322 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:17.876595 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.877391 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:17.878706 3810 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:17.891276 2035 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:17.854048 3705 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:17.882593 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.884553 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.899984 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:17.900197 2064 raft_consensus.cc:1226] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:17.900334 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:17.887375 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 129575us
I20260924 00:22:17.888798 1992 raft_consensus.cc:2379] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:17.874666 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:17.890060 2141 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 1.
I20260924 00:22:17.892023 3423 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:17.892424 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:17.893687 2136 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:17.893931 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.861701 3813 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260924 00:22:17.889055 3809 log.cc:924] Time spent T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260924 00:22:17.898531 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:17.899935 2065 raft_consensus.cc:1226] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:17.967106 2139 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:17.906713 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.956398 2143 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 6: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:17.962805 3816 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:17.964682 3733 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260924 00:22:17.965372 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 65114.3us
I20260924 00:22:17.960736 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.955569 2145 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:17.969687 1993 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:17.967231 2109 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:17.899977 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 98557.2us
I20260924 00:22:17.909564 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:17.968835 2068 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:17.910498 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:17.897476 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:17.958799 3697 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:17.991008 3820 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:17.990882 3697 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:17.991458 2136 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:17.992846 3697 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:17.999105 3819 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:17.910785 3771 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:17.999848 2137 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:17.971839 3692 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260924 00:22:17.998287 1988 raft_consensus.cc:2362] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4. Re-sending same reply.
I20260924 00:22:18.001221 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:17.911688 3683 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.006028 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:17.986435 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:17.911165 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.008286 1989 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:18.009447 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.010140 2139 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:18.011440 2065 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:18.007165 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.012600 1988 raft_consensus.cc:2379] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:18.013366 3705 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.014618 3705 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:17.909802 3810 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.017815 2110 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.018508 3810 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.018651 3692 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:18.019098 3692 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:18.019457 3692 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:18.024171 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 128) took 2034 ms. Trace:
I20260924 00:22:18.024638 1891 rpcz_store.cc:276] 0924 00:22:15.997518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:16.048229 (+ 50711us) service_pool.cc:227] Handling call
0924 00:22:18.024096 (+1975867us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:17.164325 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:17.164979 (+ 654us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:17.165044 (+ 65us) write_op.cc:435] Acquired schema lock
0924 00:22:17.165066 (+ 22us) tablet.cc:660] Decoding operations
0924 00:22:17.189523 (+ 24457us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:17.189662 (+ 139us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:17.189720 (+ 58us) tablet.cc:683] Acquiring locks for 2 operations
0924 00:22:17.190172 (+ 452us) tablet.cc:699] Row locks acquired
0924 00:22:17.190238 (+ 66us) write_op.cc:260] PREPARE: finished
0924 00:22:17.190410 (+ 172us) write_op.cc:270] Start()
0924 00:22:17.190549 (+ 139us) write_op.cc:276] Timestamp: P: 1790209337190355 usec, L: 0
0924 00:22:17.190584 (+ 35us) op_driver.cc:331] REPLICATION: starting
0924 00:22:17.191809 (+ 1225us) log.cc:842] Serialized 903 byte log entry
0924 00:22:17.399451 (+207642us) op_driver.cc:444] REPLICATION: finished
0924 00:22:17.483433 (+ 83982us) write_op.cc:301] APPLY: starting
0924 00:22:17.483529 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:17.483669 (+ 140us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:17.483791 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:17.484310 (+ 519us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:17.514147 (+ 29837us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 00:22:17.521808 (+ 7661us) write_op.cc:312] APPLY: finished
0924 00:22:17.561352 (+ 39544us) log.cc:842] Serialized 29 byte log entry
0924 00:22:17.561569 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:17.561661 (+ 92us) write_op.cc:454] Released schema lock
0924 00:22:17.561702 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:17.593796 (+ 32094us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:17.595352 (+ 1556us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34635,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":69214,"apply.run_cpu_time_us":3008,"apply.run_wall_time_us":120608,"num_ops":2,"prepare.queue_time_us":142242,"prepare.run_cpu_time_us":3611,"prepare.run_wall_time_us":36820,"replication_time_us":163097,"thread_start_us":115532,"threads_started":2}]]}
W20260924 00:22:18.034711 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 67536.9us
I20260924 00:22:18.035595 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.036214 1994 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:18.037283 2614 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:18.038300 3697 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:18.038805 3697 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.039345 3697 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.040462 3521 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:18.044479 2065 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:18.046164 2063 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:17.988392 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:17.989256 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:17.999596 3819 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:17.966948 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.001837 2138 raft_consensus.cc:2379] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:17.909682 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.002437 3724 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:18.002707 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.060715 2644 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:18.060724 3819 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.061378 1995 raft_consensus.cc:2379] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:18.003159 2066 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:18.003232 3529 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.048295 3702 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:18.064455 1993 raft_consensus.cc:2379] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:18.064803 3529 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:18.004438 2036 leader_election.cc:334] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.928s (SENT)
I20260924 00:22:18.065325 2140 raft_consensus.cc:1226] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 2. Current term is 3. Ops: []
W20260924 00:22:18.065685 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 61271.4us
W20260924 00:22:18.066356 2036 leader_election.cc:334] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.937s (SENT)
I20260924 00:22:18.066737 2036 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:17.996989 2642 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:17.987427 3423 raft_consensus.cc:686] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:18.060506 1990 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:18.070571 3823 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:18.084259 1992 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:18.047135 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.075640 2068 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
W20260924 00:22:18.089612 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 103238us
I20260924 00:22:18.073184 3820 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:18.088855 3702 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:18.092463 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.094430 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.087977 1989 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:18.097945 3692 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.111889 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:18.071602 3819 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260924 00:22:18.070623 3423 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:18.113296 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 45561.3us
I20260924 00:22:18.114117 2109 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.117548 2109 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.119112 3692 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.119246 3810 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:18.105895 1893 catalog_manager.cc:5495] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:18.109812 3697 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.120894 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.099390 3702 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:18.132346 1893 catalog_manager.cc:5495] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:18.138010 3702 raft_consensus.cc:686] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.139009 3702 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.138762 1893 catalog_manager.cc:5495] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 2 to 3, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 3 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:18.130653 2109 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.134943 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.141072 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:18.141283 3733 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:18.141783 3733 raft_consensus.cc:686] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.142999 2068 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:18.135500 2036 leader_election.cc:334] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.957s (SENT)
I20260924 00:22:18.131458 3392 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:18.137419 3692 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.122752 3705 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:18.146970 2063 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:18.146515 3733 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.151343 3804 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
W20260924 00:22:18.151901 2036 leader_election.cc:334] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.982s (SENT)
I20260924 00:22:18.152432 2036 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:18.144194 3771 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:18.152242 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.145109 3529 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:18.166846 2140 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:18.165416 1893 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:18.171684 2068 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:18.170847 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:18.169395 2644 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:18.165489 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:18.152038 3724 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260924 00:22:18.171504 3771 raft_consensus.cc:686] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.190671 2640 raft_consensus.cc:2379] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:18.176877 3392 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:18.149358 3692 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.179388 1893 catalog_manager.cc:5495] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 2, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 2 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:18.181954 3697 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.157527 3423 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:18.191737 3684 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.188104 3820 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.191565 3771 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.196707 3684 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.197326 3820 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.198762 3684 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.195564 2036 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.199210 3820 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.200115 3423 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:18.200835 2036 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.201630 3423 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:18.202683 3810 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:18.202705 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:18.203341 3810 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:18.203595 2140 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:18.211272 2144 raft_consensus.cc:1226] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:18.212919 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.212193 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.214849 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:18.215497 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d6c2f84c42b9478e954135118d17b4e2, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.215624 2137 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
W20260924 00:22:18.215811 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d6c2f84c42b9478e954135118d17b4e2, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.216817 2139 raft_consensus.cc:1261] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:18.218339 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:18.219542 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.220058 1995 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.224100 2066 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:18.224149 2140 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:18.225965 2036 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.226958 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.225678 1893 catalog_manager.cc:5495] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:18.229573 2068 raft_consensus.cc:1261] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:18.231418 2142 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260924 00:22:18.225549 3813 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260924 00:22:18.233286 3700 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:18.233914 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:18.234655 3481 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:18.235020 3481 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:18.234396 1893 catalog_manager.cc:5495] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:18.235381 3481 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:18.221769 3423 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260924 00:22:18.236057 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.236162 3816 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:18.223618 3819 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260924 00:22:18.237344 3705 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:18.237779 3705 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.238593 3705 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.242009 3702 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:18.243883 3692 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.244350 3801 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:18.244398 3830 log.cc:924] Time spent T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260924 00:22:18.244926 3801 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.245517 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:18.246901 3801 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.248829 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:18.249385 2140 raft_consensus.cc:595] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.249697 2140 raft_consensus.cc:603] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:18.249856 2144 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:18.250030 3692 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:18.251777 2109 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:18.251838 2139 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260924 00:22:18.253059 3832 log.cc:924] Time spent T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.080s user 0.003s sys 0.000s
I20260924 00:22:18.253564 3692 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.217281 3810 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.255988 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.256640 3810 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:18.257048 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.257716 1991 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:18.256592 1994 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:18.261992 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.262682 3801 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:18.263008 3801 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.263310 3801 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.264901 2110 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.251912 3481 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.267328 3820 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:18.273378 3820 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.273806 3820 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:18.275177 3807 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:18.276070 2035 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:18.278393 3683 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:18.278827 3683 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.279259 3683 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:18.286971 3813 consensus_queue.cc:1046] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:18.287842 3819 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260924 00:22:18.291884 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 65969.6us
I20260924 00:22:18.292799 2036 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.297399 2137 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:18.300436 1995 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:18.301743 2614 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:18.304078 3697 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:18.304593 3697 raft_consensus.cc:686] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:18.305406 3697 consensus_queue.cc:237] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.309384 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:18.288520 3391 raft_consensus.cc:3032] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:18.312862 3807 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:18.314039 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:18.314172 3423 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:18.314810 2036 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.315351 3423 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:18.299530 3481 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.315860 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:18.312109 3391 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:18.340168 3546 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260924 00:22:18.342696 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:18.340287 3807 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:18.343912 3807 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.346123 3807 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:18.353348 2036 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:18.354362 3546 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:18.355676 2035 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:18.319373 3813 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:18.359644 2137 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 5
I20260924 00:22:18.360713 2036 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.346745 3391 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.361471 3823 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:18.361840 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.361840 3391 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:18.359668 3807 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:18.368613 3823 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:18.315868 3423 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.370398 2137 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:18.370427 3423 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:18.374468 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:18.374949 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:18.375291 2144 raft_consensus.cc:2379] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:18.375600 2642 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:18.375813 3820 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.379783 3692 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:22:18.380832 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 143929us
I20260924 00:22:18.382261 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.401031 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:18.402065 2642 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:18.401311 3813 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:18.401832 3771 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:18.398240 3481 consensus_queue.cc:1046] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260924 00:22:18.398080 3733 consensus_queue.cc:1046] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260924 00:22:18.404721 3823 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:18.406124 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:18.406985 2036 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:18.408186 2036 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.387769 3546 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:18.413367 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:18.414388 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.389002 3807 raft_consensus.cc:686] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:18.416317 3807 consensus_queue.cc:237] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.395371 3819 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:18.421550 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:18.422734 2144 raft_consensus.cc:2379] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:18.425208 3756 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:18.429832 3820 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.430315 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.413506 3655 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:18.430498 3482 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:18.431432 3391 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:18.431953 3482 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:18.432734 3481 consensus_queue.cc:1225] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 8c129c01e1c047628f6a87b001aa7258 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260924 00:22:18.432873 3423 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:18.433314 3423 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:18.391789 3683 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.433722 3423 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.434310 3733 raft_consensus.cc:3032] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:18.434705 3733 raft_consensus.cc:729] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.433472 3700 raft_consensus.cc:3032] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:18.437067 1995 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:18.430326 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.438102 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:18.446197 2144 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:18.438032 3612 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.435458 3733 consensus_queue.cc:260] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.436110 3813 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:18.447232 3612 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.447819 3813 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:18.448271 3813 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:18.433329 3481 consensus_queue.cc:1046] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260924 00:22:18.449250 3612 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.449524 3481 raft_consensus.cc:3032] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:18.449877 3481 raft_consensus.cc:729] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.450722 3481 consensus_queue.cc:260] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.453167 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.453768 2064 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:18.432734 3796 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260924 00:22:18.455614 3771 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:18.438666 3700 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.447702 3733 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:18.458278 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.461633 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.461720 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:18.462320 2067 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.435449 3683 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.466686 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.467440 2068 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:18.467913 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.463845 3837 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:18.474140 3837 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:18.467599 3702 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:18.475546 3702 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:18.476025 3702 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:18.477461 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:18.481380 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.481995 2071 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:18.451807 3481 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:18.483762 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:18.485116 2066 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:18.486065 2144 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:18.460115 3771 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.490254 3771 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.492590 2035 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:18.486619 3845 log.cc:924] Time spent T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260924 00:22:18.496855 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:18.456867 3482 consensus_queue.cc:1159] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "948356bf66a941ef8249ecbd0ec94f6d" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260924 00:22:18.463112 3705 consensus_queue.cc:1046] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260924 00:22:18.496482 3546 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:18.498862 3810 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.463332 3700 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.529522 1995 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:18.531183 2110 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.525668 3800 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.532133 3800 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.534380 3800 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:18.536578 3702 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.539908 3800 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:18.540647 3816 consensus_queue.cc:1046] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:18.539839 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:18.512283 3692 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.542305 2642 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:18.542490 3692 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.544624 3692 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.517563 3771 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.545681 3705 raft_consensus.cc:3032] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:18.546058 3705 raft_consensus.cc:729] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:18.546880 3705 consensus_queue.cc:260] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.548007 3705 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:18.553386 3800 raft_consensus.cc:686] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:18.554517 3800 consensus_queue.cc:237] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.530081 3810 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.557701 3810 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.558415 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.559062 2066 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:18.561698 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.562306 2068 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:18.563473 2109 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:18.564266 3479 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:18.564666 3479 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.565054 3479 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.538965 3546 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:18.568400 3546 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.564258 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.569949 2068 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:18.571171 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.571920 2066 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:18.579005 3700 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.602012 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.603150 1991 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
W20260924 00:22:18.611202 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 148131us
I20260924 00:22:18.612238 2067 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:18.611475 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.613709 2110 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.615033 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.619992 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:18.621065 2067 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:18.625654 3850 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:18.630029 2067 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:18.633149 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.633333 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.627084 3692 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.634142 2067 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:18.554551 3801 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.633167 3684 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:18.645663 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.653523 2066 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:18.655797 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:18.628312 3816 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
W20260924 00:22:18.669668 3842 log.cc:924] Time spent T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260924 00:22:18.671811 3702 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.667084 3479 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.676190 3801 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.678583 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.705423 1995 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.683074 3392 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:18.683490 2109 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:18.691207 2138 raft_consensus.cc:2379] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:18.692518 1890 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:18.653213 3482 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:18.653007 3684 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.715126 2109 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:18.711226 2071 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:18.732656 1995 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:18.643607 3692 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.729799 1890 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 3 to 4, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 4 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:18.742874 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.745973 2109 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:18.726810 3702 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.739156 3392 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:18.739323 3684 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:18.738824 3683 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.744932 3801 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260924 00:22:18.758915 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 143950us
I20260924 00:22:18.753234 3529 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:18.753583 3392 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:18.755983 3521 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:18.768368 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.775880 2109 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:18.776336 2066 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
W20260924 00:22:18.769012 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 114152us
I20260924 00:22:18.777328 2067 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:18.772981 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33232 (request call id 216) took 2639 ms. Trace:
I20260924 00:22:18.768603 3850 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260924 00:22:18.770874 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:18.775256 2068 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:18.778508 2139 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:18.779436 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.781291 2140 raft_consensus.cc:1226] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:18.782217 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:18.787294 1991 raft_consensus.cc:2379] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:18.780849 2648 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:18.787796 2071 raft_consensus.cc:2379] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:18.753826 3702 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.820662 2145 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:18.775949 3392 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:18.824308 2068 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:18.820616 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.781068 3771 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.756599 3683 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.838333 3803 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
I20260924 00:22:18.830739 1994 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260924 00:22:18.840526 3683 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:18.842271 3857 log.cc:924] Time spent T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260924 00:22:18.842414 3702 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.829696 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.843381 1988 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260924 00:22:18.844934 3853 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260924 00:22:18.848505 3692 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:18.778453 3521 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.849473 3521 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:18.850145 2642 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:18.851292 3479 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:18.829576 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:18.858021 2136 raft_consensus.cc:2379] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:18.859001 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:18.832942 2067 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:18.859737 2136 raft_consensus.cc:2354] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:18.790786 3529 raft_consensus.cc:686] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:18.860845 3529 consensus_queue.cc:237] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.838148 3771 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.838434 3858 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260924 00:22:18.863225 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.870877 1994 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:18.838375 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:18.768844 3816 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260924 00:22:18.868453 1987 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:18.872068 2145 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:18.879208 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.879906 1995 raft_consensus.cc:2379] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:18.769502 3423 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.778311 1893 rpcz_store.cc:276] 0924 00:22:16.122148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:16.122345 (+ 197us) service_pool.cc:227] Handling call
0924 00:22:18.761386 (+2639041us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:18.320721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:18.321394 (+ 673us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:18.321426 (+ 32us) write_op.cc:435] Acquired schema lock
0924 00:22:18.321449 (+ 23us) tablet.cc:660] Decoding operations
0924 00:22:18.322419 (+ 970us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:18.322506 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:18.322545 (+ 39us) tablet.cc:683] Acquiring locks for 7 operations
0924 00:22:18.323316 (+ 771us) tablet.cc:699] Row locks acquired
0924 00:22:18.323341 (+ 25us) write_op.cc:260] PREPARE: finished
0924 00:22:18.323538 (+ 197us) write_op.cc:270] Start()
0924 00:22:18.323669 (+ 131us) write_op.cc:276] Timestamp: P: 1790209338323486 usec, L: 0
0924 00:22:18.323697 (+ 28us) op_driver.cc:331] REPLICATION: starting
0924 00:22:18.324495 (+ 798us) log.cc:842] Serialized 2828 byte log entry
0924 00:22:18.520559 (+196064us) op_driver.cc:444] REPLICATION: finished
0924 00:22:18.660182 (+139623us) write_op.cc:301] APPLY: starting
0924 00:22:18.660290 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:18.660559 (+ 269us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:18.660737 (+ 178us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:18.662755 (+ 2018us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:18.668043 (+ 5288us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0924 00:22:18.668124 (+ 81us) write_op.cc:312] APPLY: finished
0924 00:22:18.668440 (+ 316us) log.cc:842] Serialized 59 byte log entry
0924 00:22:18.669785 (+ 1345us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:18.669952 (+ 167us) write_op.cc:454] Released schema lock
0924 00:22:18.670018 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:18.672621 (+ 2603us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:18.672651 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8996,"reported_tablets":34,"child_traces":[["op",{"apply.queue_time_us":131522,"apply.run_cpu_time_us":3845,"apply.run_wall_time_us":12892,"num_ops":7,"prepare.queue_time_us":49522,"prepare.run_cpu_time_us":4067,"prepare.run_wall_time_us":4067,"replication_time_us":186394,"thread_start_us":162342,"threads_started":2}]]}
I20260924 00:22:18.860958 2136 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:18.890219 2140 raft_consensus.cc:1226] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:18.877538 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:18.891080 1991 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:18.892479 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 116631us
I20260924 00:22:18.893158 2110 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:18.894241 2109 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:18.896710 1992 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260924 00:22:18.897627 3655 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.901062 3655 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:18.879006 3803 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.905992 2145 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260924 00:22:18.915099 3860 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20260924 00:22:18.916785 3771 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.917699 2644 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260924 00:22:18.918422 3423 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:18.920807 1993 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:18.836760 3813 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.912498 3810 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:18.912765 3800 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:18.944196 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:18.949350 2064 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:18.956061 3642 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260924 00:22:18.956820 3849 log.cc:924] Time spent T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
I20260924 00:22:18.944249 3810 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:18.960315 3810 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:18.961481 2640 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:18.966964 2036 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:18.947045 3800 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:18.912883 3803 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:18.981462 2068 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:18.947394 3655 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:18.990535 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.990942 1996 raft_consensus.cc:1261] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:18.996410 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:18.986878 3801 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:18.987926 3800 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:19.005520 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 65849.1us
I20260924 00:22:18.995326 3423 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:19.015419 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:19.016696 3683 consensus_queue.cc:1046] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260924 00:22:19.028880 3872 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:18.998771 3593 consensus_queue.cc:1035] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260924 00:22:18.999217 3823 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260924 00:22:19.041463 1960 leader_election.cc:398] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:19.049890 1960 leader_election.cc:401] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260924 00:22:19.043478 3858 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:19.043674 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:19.044466 3864 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260924 00:22:19.022745 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:19.034475 3392 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.025300 3820 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:19.045043 3865 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
W20260924 00:22:19.045964 3868 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260924 00:22:19.028077 3813 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:19.061980 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:19.047173 3684 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.066793 3820 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.072817 3801 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:19.027704 3816 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260924 00:22:19.034279 3756 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
W20260924 00:22:19.103924 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 209722us
W20260924 00:22:19.109845 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 202002us
I20260924 00:22:19.108686 2109 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:19.073837 3733 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:19.069249 1990 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:19.079932 3801 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:19.109875 3850 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260924 00:22:19.044924 3872 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:19.141369 3803 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:19.142167 3871 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:19.142635 3871 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:19.133075 3733 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:19.144266 3733 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:19.144645 3871 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:19.109375 3702 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260924 00:22:19.157974 1991 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:19.159864 3771 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
W20260924 00:22:19.161821 3877 log.cc:924] Time spent T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260924 00:22:19.162179 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:19.136391 3521 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.169262 3521 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.170014 2143 raft_consensus.cc:1226] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260924 00:22:19.171289 3684 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:19.156168 3705 consensus_meta.cc:287] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.608s user 0.009s sys 0.002s
I20260924 00:22:19.173171 3392 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.175482 3820 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.159924 3837 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260924 00:22:19.155153 3529 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election won for term 2
W20260924 00:22:19.196162 3873 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.240s user 0.000s sys 0.003s
I20260924 00:22:19.156096 3810 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.201833 3529 raft_consensus.cc:686] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:19.213913 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:19.215715 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:19.216310 1987 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:19.219182 3866 log.cc:924] Time spent T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.255s user 0.002s sys 0.000s
W20260924 00:22:19.219352 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 189678us
I20260924 00:22:19.220079 3683 raft_consensus.cc:3032] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:19.220588 3683 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:19.221400 3683 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.222617 3683 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:19.226238 3391 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:19.229945 3823 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:19.231036 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 129) took 1183 ms. Trace:
I20260924 00:22:19.220013 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:19.234099 1987 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:19.243848 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:19.246146 2110 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:19.205732 3801 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.253093 1891 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:19.255391 3479 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
W20260924 00:22:19.263805 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 155247us
I20260924 00:22:19.267122 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.267699 2068 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:19.231403 1890 rpcz_store.cc:276] 0924 00:22:18.047532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:18.047769 (+ 237us) service_pool.cc:227] Handling call
0924 00:22:19.230979 (+1183210us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:18.775145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:18.902206 (+127061us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:18.902255 (+ 49us) write_op.cc:435] Acquired schema lock
0924 00:22:18.902278 (+ 23us) tablet.cc:660] Decoding operations
0924 00:22:18.902716 (+ 438us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:18.902772 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:18.902797 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0924 00:22:18.903145 (+ 348us) tablet.cc:699] Row locks acquired
0924 00:22:18.903163 (+ 18us) write_op.cc:260] PREPARE: finished
0924 00:22:18.903311 (+ 148us) write_op.cc:270] Start()
0924 00:22:18.903421 (+ 110us) write_op.cc:276] Timestamp: P: 1790209338903269 usec, L: 0
0924 00:22:18.903442 (+ 21us) op_driver.cc:331] REPLICATION: starting
0924 00:22:18.904001 (+ 559us) log.cc:842] Serialized 907 byte log entry
0924 00:22:19.004538 (+100537us) op_driver.cc:444] REPLICATION: finished
0924 00:22:19.028032 (+ 23494us) write_op.cc:301] APPLY: starting
0924 00:22:19.037218 (+ 9186us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:19.049231 (+ 12013us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:19.064204 (+ 14973us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:19.073561 (+ 9357us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:19.079271 (+ 5710us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 00:22:19.079345 (+ 74us) write_op.cc:312] APPLY: finished
0924 00:22:19.112236 (+ 32891us) log.cc:842] Serialized 29 byte log entry
0924 00:22:19.140787 (+ 28551us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:19.150610 (+ 9823us) write_op.cc:454] Released schema lock
0924 00:22:19.155156 (+ 4546us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:19.176683 (+ 21527us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:19.176714 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":29674,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":9546,"apply.run_cpu_time_us":2840,"apply.run_wall_time_us":150376,"num_ops":2,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":3035,"prepare.run_wall_time_us":143719,"replication_time_us":95620}]]}
I20260924 00:22:19.267917 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:19.270382 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 130779us
I20260924 00:22:19.271284 2062 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:19.267838 1891 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:19.272567 2035 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:19.274755 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.275918 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:19.274793 1891 catalog_manager.cc:5495] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 3 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 5 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:19.278551 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:19.279965 3702 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:19.280297 2069 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:19.280431 2064 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:19.281490 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:19.281711 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.282021 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:19.282682 2063 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:19.282753 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.283332 2067 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:19.285298 3392 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:19.285840 3392 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.287899 3392 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.288455 3700 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:19.289297 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:19.291141 3733 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:19.293627 3882 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260924 00:22:19.296674 3801 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.298262 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:19.293066 3733 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:19.303568 2068 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:19.304792 3797 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:19.272866 3479 raft_consensus.cc:686] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:19.305296 2067 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:19.295058 2062 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:19.306190 3479 consensus_queue.cc:237] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.306856 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.306717 3771 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:19.307574 3837 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:19.310185 1960 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.312391 1891 catalog_manager.cc:5495] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:19.314035 1891 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:19.315636 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:19.317973 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:19.318276 3872 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:19.319268 3771 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:19.309415 3733 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260924 00:22:19.324884 1987 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:19.301223 3800 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.319550 1891 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:19.320981 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:19.266635 3529 consensus_queue.cc:237] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.344455 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:19.346418 2143 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:19.348944 2137 raft_consensus.cc:2379] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:19.349686 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:19.350186 2141 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:19.349098 1891 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:19.351099 1891 catalog_manager.cc:5495] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:19.370371 3820 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260924 00:22:19.370597 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:19.340673 2640 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:19.376740 2036 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:19.333045 2643 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:19.379388 2138 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:19.380095 2142 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:19.369226 2613 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:19.392132 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.392795 2062 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:19.393841 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a4917c5d11b34f7aaa5cbdcf98709f22, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:19.394148 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet a4917c5d11b34f7aaa5cbdcf98709f22, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.395272 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:19.396021 2141 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:19.345008 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:19.333783 3810 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.343748 3800 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.344138 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:19.324586 3837 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.071s
I20260924 00:22:19.412191 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:19.425791 2143 raft_consensus.cc:1226] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:19.413956 1891 catalog_manager.cc:5495] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 3 to 4, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 4 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:19.426028 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:19.416235 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:19.427793 1996 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:19.421317 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:19.430728 2141 raft_consensus.cc:2354] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:19.447844 1993 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260924 00:22:19.455631 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.456662 3705 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:19.459359 3705 consensus_queue.cc:1046] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:19.465600 3894 raft_consensus.cc:3032] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:19.466038 3894 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:19.466828 3894 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.468047 3894 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:19.470059 3733 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:19.471304 3858 raft_consensus.cc:3032] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:19.471728 3858 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:19.472678 3858 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.473652 2062 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:19.474671 3858 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:19.484943 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 72873.8us
I20260924 00:22:19.401010 3756 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:19.492554 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:19.493781 2143 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:19.408583 3871 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:19.503332 2139 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260924 00:22:19.503398 3481 consensus_queue.cc:1046] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260924 00:22:19.499040 3867 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260924 00:22:19.509591 1992 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:19.417691 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:19.517724 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:19.424346 3683 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:19.524053 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.524650 2068 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:19.521792 1996 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:19.528609 2110 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:19.491967 3819 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:19.529687 2109 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.492542 3756 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:19.544373 3702 raft_consensus.cc:3032] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:19.544940 3702 raft_consensus.cc:729] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:19.546159 3702 consensus_queue.cc:260] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.554450 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.555522 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:19.535039 2614 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:19.502506 3871 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:19.581791 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 46889.7us
I20260924 00:22:19.520905 3813 raft_consensus.cc:2741] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260924 00:22:19.529505 2138 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:19.496248 3700 consensus_queue.cc:1046] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.089s
W20260924 00:22:19.603524 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 73901.3us
I20260924 00:22:19.553862 3804 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260924 00:22:19.579000 3521 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:19.556901 3756 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:19.608683 2109 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.587395 3871 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:19.611475 2068 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:19.629019 2109 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:19.634183 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 46581.8us
I20260924 00:22:19.643646 2035 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:19.537192 3683 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.650034 2109 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:19.651491 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.625236 3820 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:19.625566 3858 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:19.662958 2109 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:19.665949 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 170747us
I20260924 00:22:19.643071 3479 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:19.669462 2109 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.629756 3692 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260924 00:22:19.647845 3521 raft_consensus.cc:686] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:19.663695 3724 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:19.664279 3684 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:19.659561 3392 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:19.687927 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:19.697407 2139 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:19.681670 3920 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:19.681201 3810 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:19.682399 3529 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:19.705663 3655 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:19.706369 3683 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:19.706435 3919 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
W20260924 00:22:19.707636 3893 log.cc:924] Time spent T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20260924 00:22:19.703562 3920 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:19.700670 3392 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:19.667495 3858 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.703686 3837 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:19.701663 3863 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:19.710628 3684 raft_consensus.cc:686] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:19.726926 3850 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.007s
I20260924 00:22:19.734304 3863 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.734864 3858 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:19.735419 3907 log.cc:924] Time spent T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260924 00:22:19.736244 3837 raft_consensus.cc:686] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:19.724828 3392 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:19.721513 3920 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:19.737196 3837 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.742131 3521 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.742996 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:19.736948 3863 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:19.774562 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:19.776826 2070 raft_consensus.cc:2379] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:19.798535 2145 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:19.861936 3927 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:19.862210 3684 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:19.862545 3927 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:19.797824 3858 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:19.804152 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:19.822302 3873 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.090s user 0.005s sys 0.000s
I20260924 00:22:19.864151 3819 raft_consensus.cc:686] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:19.941025 3819 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:19.946877 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 194318us
I20260924 00:22:19.947957 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:19.950294 2145 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:19.967979 2138 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:19.968228 1987 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260924 00:22:19.955999 1990 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:19.975234 3871 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:19.996001 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:19.993597 2642 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260924 00:22:20.007876 2110 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:19.994307 3700 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:19.938915 3858 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.003899 3871 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.008361 3479 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260924 00:22:20.046075 3932 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:20.048417 3810 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:20.118808 3858 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.109238 3529 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.113458 3927 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.139573 3810 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:20.149822 3810 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:20.162181 3756 consensus_meta.cc:287] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.508s user 0.010s sys 0.000s
I20260924 00:22:20.163548 3756 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.135030 3920 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:20.175894 2035 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.012s sys 0.034s
W20260924 00:22:20.210398 2035 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.012s sys 0.034s
I20260924 00:22:20.139230 3392 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:20.217075 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 563588us
I20260924 00:22:20.155382 3529 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.141702 3932 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:20.224035 2068 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.024s sys 0.002s
I20260924 00:22:20.216434 3756 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:20.252971 2615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.019s sys 0.054s
W20260924 00:22:20.253343 2615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.019s sys 0.054s
I20260924 00:22:20.254518 2035 leader_election.cc:398] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:20.257104 3724 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.260623 3724 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.261173 3807 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260924 00:22:20.262255 3910 log.cc:924] Time spent T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260924 00:22:20.262639 3724 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:20.247902 3858 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.252045 3928 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.258857 3756 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20260924 00:22:20.228704 2068 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.541s user 0.025s sys 0.002s
I20260924 00:22:20.281988 3797 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:20.380255 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 88369.2us
I20260924 00:22:20.283159 3920 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.274639 3392 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:20.275946 3529 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:20.396862 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.004s sys 0.037s
W20260924 00:22:20.399458 2069 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.051s sys 0.055s
I20260924 00:22:20.277667 3932 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:20.397312 2109 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.020s sys 0.015s
W20260924 00:22:20.408165 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.004s sys 0.037s
W20260924 00:22:20.406569 2613 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.012s sys 0.020s
W20260924 00:22:20.408521 2109 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.020s sys 0.015s
W20260924 00:22:20.408538 2069 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.750s user 0.051s sys 0.056s
W20260924 00:22:20.408842 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 718187us
W20260924 00:22:20.413491 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 744064us
W20260924 00:22:20.409036 2613 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.725s user 0.012s sys 0.020s
I20260924 00:22:20.414568 2109 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:20.419999 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:20.401264 3807 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.300594 3858 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.449889 3932 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:20.450207 3933 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:20.450455 3932 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:20.450814 3933 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.452843 3932 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:20.300834 3928 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.442734 3392 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:20.453665 3931 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.454315 1988 raft_consensus.cc:1261] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:20.455392 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:20.456411 3733 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.444334 3683 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:20.394891 3797 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.458889 3933 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.449791 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:20.460882 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:20.473121 3928 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.485792 2640 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:20.485638 3683 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.487066 2069 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:20.487314 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:20.487812 3683 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.487911 2070 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:20.488974 1994 raft_consensus.cc:2396] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260924 00:22:20.488832 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 57719.3us
I20260924 00:22:20.490660 2070 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260924 00:22:20.490792 1991 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.034s sys 0.048s
W20260924 00:22:20.494196 1991 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.779s user 0.035s sys 0.048s
I20260924 00:22:20.503850 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:20.504714 2062 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
W20260924 00:22:20.493378 3684 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.015s sys 0.010s
I20260924 00:22:20.510545 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:20.502377 1960 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.035s sys 0.031s
W20260924 00:22:20.518327 1960 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.569s user 0.035s sys 0.031s
W20260924 00:22:20.507910 3684 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.015s sys 0.010s
W20260924 00:22:20.521432 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 573544us
W20260924 00:22:20.522260 3612 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.017s sys 0.029s
W20260924 00:22:20.522586 3612 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.017s sys 0.029s
W20260924 00:22:20.524233 2063 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.004s sys 0.022s
W20260924 00:22:20.530879 2063 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.873s user 0.004s sys 0.022s
I20260924 00:22:20.531574 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1134) took 1122 ms. Trace:
W20260924 00:22:20.532100 3898 log.cc:924] Time spent T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.877s user 0.023s sys 0.000s
I20260924 00:22:20.535293 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:20.550330 2071 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.016s sys 0.037s
I20260924 00:22:20.550482 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1289) took 1282 ms. Trace:
W20260924 00:22:20.551784 2071 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.836s user 0.016s sys 0.037s
W20260924 00:22:20.553843 3894 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.005s sys 0.043s
I20260924 00:22:20.557408 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:20.561616 2141 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:20.475073 3807 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.558120 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:20.562563 1995 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:20.488828 3733 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.562815 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.490010 3810 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.551823 2065 rpcz_store.cc:276] 0924 00:22:19.268339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.268468 (+ 129us) service_pool.cc:227] Handling call
0924 00:22:19.270761 (+ 2293us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:19.273646 (+ 2885us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:19.273674 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:19.302117 (+ 28443us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b2c000002e8. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d349fe11 00007fb6d34af442 00007fb6d4af7c11 00007fb6d4abc58a 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772
0924 00:22:19.302508 (+ 391us) log.cc:842] Serialized 26 byte log entry
0924 00:22:19.430989 (+128481us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:19.598168 (+167179us) log.cc:842] Serialized 14 byte log entry
0924 00:22:19.647814 (+ 49646us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:19.647917 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:20.267041 (+619124us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b18002ee5e0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d1b61f96 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511
0924 00:22:20.531478 (+264437us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:20.550275 (+ 18797us) raft_consensus.cc:1671] finished
0924 00:22:20.550295 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:20.550430 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54389,"prepare.queue_time_us":317357,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":29865,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":13556,"spinlock_wait_cycles":36115456,"thread_start_us":229562,"threads_started":2,"wal-append.queue_time_us":287852}
I20260924 00:22:20.564347 2613 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:20.565412 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:20.566865 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 48802.1us
I20260924 00:22:20.527664 3702 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:22:20.563441 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.951s user 0.011s sys 0.045s
I20260924 00:22:20.571931 1996 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
W20260924 00:22:20.573937 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.013s user 0.012s sys 0.045s
I20260924 00:22:20.572642 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:20.531909 2063 rpcz_store.cc:276] 0924 00:22:19.409470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.409679 (+ 209us) service_pool.cc:227] Handling call
0924 00:22:19.488505 (+ 78826us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:19.488681 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 3
0924 00:22:19.654567 (+165886us) log.cc:842] Serialized 14 byte log entry
0924 00:22:20.531199 (+876632us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:20.531266 (+ 67us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:20.531322 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:20.531404 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:20.531526 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11689,"prepare.queue_time_us":259875,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":41197,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":10623,"thread_start_us":997720,"threads_started":2}
I20260924 00:22:20.574869 2069 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:20.572642 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:20.580626 3929 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.000s sys 0.022s
I20260924 00:22:20.581043 2646 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:20.581904 3810 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:20.566251 3894 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s user 0.005s sys 0.043s
I20260924 00:22:20.582826 2062 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:20.583350 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:20.589322 3895 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.589919 3895 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.591254 2144 raft_consensus.cc:2379] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:20.591084 3858 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.592185 3895 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:20.593314 3807 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.593618 3820 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:20.594877 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.595150 3879 consensus_queue.cc:1035] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:20.595643 2145 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:20.596664 3940 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:20.597707 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.598937 2138 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:20.599050 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:20.599740 1989 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 6
I20260924 00:22:20.600981 3733 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.601643 3733 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:20.602206 3733 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.603549 2141 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:20.609448 1989 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:20.605573 1995 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:20.478785 3931 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.611730 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:20.613615 1995 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:20.615024 3931 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.617033 2110 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:20.617471 3797 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.618021 3810 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:20.619288 2070 raft_consensus.cc:1261] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 00:22:20.619390 3810 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:20.620042 3810 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:20.582525 3929 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.000s sys 0.023s
W20260924 00:22:20.639947 3898 log.cc:924] Time spent T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260924 00:22:20.578368 3890 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:20.641746 2109 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:20.641840 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:20.648653 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:20.657960 2064 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.013s sys 0.018s
I20260924 00:22:20.640074 3481 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:20.659196 2064 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.931s user 0.013s sys 0.019s
I20260924 00:22:20.592839 3895 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
W20260924 00:22:20.642403 3875 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.013s sys 0.010s
I20260924 00:22:20.651793 3733 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.680883 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:20.673247 3771 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.014s sys 0.049s
I20260924 00:22:20.684342 2109 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:20.666672 3482 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.009s sys 0.021s
W20260924 00:22:20.686458 3875 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.013s sys 0.010s
I20260924 00:22:20.683495 3863 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260924 00:22:20.707454 3771 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.014s sys 0.049s
I20260924 00:22:20.723047 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:20.649860 3890 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:20.715090 3482 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.593s user 0.009s sys 0.021s
I20260924 00:22:20.731716 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:20.672713 3481 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.726267 1995 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:20.710335 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:20.747048 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:20.724620 3863 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:20.777529 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 49944.1us
I20260924 00:22:20.747014 3883 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:20.748013 3423 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:20.750882 3863 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:20.798180 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:20.802906 2136 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.053s user 0.026s sys 0.019s
I20260924 00:22:20.788661 3771 raft_consensus.cc:3032] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:20.810932 1960 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:20.809860 2136 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.060s user 0.026s sys 0.019s
W20260924 00:22:20.776731 3837 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.006s sys 0.022s
W20260924 00:22:20.819509 3837 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s user 0.006s sys 0.023s
I20260924 00:22:20.807556 3733 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.858307 3697 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:20.916930 3733 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.915176 3684 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:20.915300 3807 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:20.918584 3684 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:20.919196 3733 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:20.919960 3733 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:20.920296 3733 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:20.920399 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1138) took 1222 ms. Trace:
I20260924 00:22:20.920683 3733 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:20.814647 3771 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:20.866032 2036 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:20.921432 3700 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:20.815724 3482 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260924 00:22:20.921955 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 56316.2us
I20260924 00:22:20.922165 3482 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:20.920780 2071 rpcz_store.cc:276] 0924 00:22:19.697775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.697941 (+ 166us) service_pool.cc:227] Handling call
0924 00:22:19.714047 (+ 16106us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:19.714454 (+ 407us) raft_consensus.cc:1520] Early marking committed up to index 2
0924 00:22:19.714484 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:19.714913 (+ 429us) log.cc:842] Serialized 26 byte log entry
0924 00:22:20.552255 (+837342us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:20.552292 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:20.552357 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:20.920185 (+367828us) raft_consensus.cc:1671] finished
0924 00:22:20.920215 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:20.920349 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24102,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":9394,"thread_start_us":836646,"threads_started":1,"wal-append.queue_time_us":1102975}
I20260924 00:22:20.922621 3482 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:20.923041 2036 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:20.819350 3936 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:20.797317 3423 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:20.871651 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:20.873899 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:20.880363 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.925807 2063 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:20.813146 3950 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260924 00:22:20.909482 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.934127 2141 raft_consensus.cc:2396] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260924 00:22:20.908846 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:20.917037 3697 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:20.810652 3940 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:20.908818 3951 consensus_queue.cc:1046] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.009s
I20260924 00:22:20.918892 3807 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:20.795522 2643 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.075s user 0.002s sys 0.033s
W20260924 00:22:20.927814 3944 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260924 00:22:20.927316 3936 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:20.928273 3924 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:20.929642 2036 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:20.863181 2068 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.009s sys 0.015s
I20260924 00:22:20.979684 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.922039 3700 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:20.980367 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.920848 3684 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:20.981236 2068 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.009s sys 0.018s
I20260924 00:22:20.955277 2065 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:20.988687 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:20.989498 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:20.989753 3697 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:20.992499 1995 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:20.937966 3612 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:20.985864 3890 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:20.993562 3937 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:20.994524 3936 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:20.995026 3938 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:20.941075 3481 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:20.998443 3952 log.cc:924] Time spent T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260924 00:22:21.002545 3807 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.002914 3391 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:21.032599 3938 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.033306 3391 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:21.033895 3391 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:21.003355 2069 raft_consensus.cc:2396] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260924 00:22:21.034773 3771 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.035414 3937 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.037165 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:20.981455 2643 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.262s user 0.012s sys 0.039s
I20260924 00:22:21.011436 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1135) took 1325 ms. Trace:
I20260924 00:22:20.980006 3924 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:21.014926 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:21.009860 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:21.018043 2138 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:20.952818 3940 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.026172 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1211) took 1650 ms. Trace:
I20260924 00:22:21.026459 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1351) took 1333 ms. Trace:
I20260924 00:22:21.076027 1991 rpcz_store.cc:276] 0924 00:22:19.693086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.693375 (+ 289us) service_pool.cc:227] Handling call
0924 00:22:19.693823 (+ 448us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:19.694263 (+ 440us) raft_consensus.cc:1520] Early marking committed up to index 1
0924 00:22:19.694305 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:19.694823 (+ 518us) log.cc:842] Serialized 26 byte log entry
0924 00:22:20.494705 (+799882us) raft_consensus.cc:1631] Marking committed up to 1
0924 00:22:20.494757 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:20.494842 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:21.026160 (+531318us) raft_consensus.cc:1671] finished
0924 00:22:21.026191 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.026365 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1525,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":6501,"thread_start_us":779008,"threads_started":1,"wal-append.queue_time_us":1019415}
I20260924 00:22:21.039222 2145 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:21.061223 2068 rpcz_store.cc:276] 0924 00:22:19.686115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.686428 (+ 313us) service_pool.cc:227] Handling call
0924 00:22:19.686728 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:19.686919 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 4
0924 00:22:20.295082 (+608163us) log.cc:842] Serialized 14 byte log entry
0924 00:22:21.011111 (+716029us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:21.011163 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:21.011206 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:21.011272 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.011383 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4639,"prepare.queue_time_us":63969,"prepare.run_cpu_time_us":2292,"prepare.run_wall_time_us":2719,"raft.queue_time_us":4848,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":29415,"thread_start_us":1318560,"threads_started":2,"wal-append.queue_time_us":731983}
W20260924 00:22:21.072974 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 63192.1us
I20260924 00:22:21.087217 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:21.088047 3593 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:21.088364 3837 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:21.026170 3612 raft_consensus.cc:686] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
W20260924 00:22:21.040613 1988 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.011s sys 0.010s
W20260924 00:22:21.092952 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.011s sys 0.025s
I20260924 00:22:21.073904 3924 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:22:21.095616 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.012s sys 0.025s
I20260924 00:22:21.075773 3816 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:21.076262 3771 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:21.097074 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:21.076372 3936 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:21.103734 2139 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.010s sys 0.009s
I20260924 00:22:21.074901 3529 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:21.107180 2110 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:21.108355 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:21.118846 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:21.084553 3521 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.023s sys 0.030s
W20260924 00:22:21.162252 3521 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s user 0.024s sys 0.031s
W20260924 00:22:21.119741 1996 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.004s sys 0.016s
W20260924 00:22:21.163882 1996 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s user 0.004s sys 0.017s
I20260924 00:22:21.084467 3869 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:21.155848 2144 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:21.092119 3801 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:21.156977 2139 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.563s user 0.012s sys 0.010s
I20260924 00:22:21.095659 3807 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:21.149677 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1137) took 1453 ms. Trace:
I20260924 00:22:21.161366 3936 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:21.101013 1988 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.011s sys 0.011s
I20260924 00:22:21.163097 3756 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:21.156909 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 468682us
I20260924 00:22:21.168141 3938 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.168418 3801 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:21.169474 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 62384.9us
I20260924 00:22:21.105229 3816 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:21.169999 3837 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:21.120466 3951 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:21.091883 3927 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260924 00:22:21.178454 3521 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:21.172420 2064 rpcz_store.cc:276] 0924 00:22:19.696085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.696247 (+ 162us) service_pool.cc:227] Handling call
0924 00:22:19.711013 (+ 14766us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:19.711224 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 3
0924 00:22:20.682961 (+971737us) log.cc:842] Serialized 14 byte log entry
0924 00:22:21.120821 (+437860us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:21.149367 (+ 28546us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:21.149419 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:21.149491 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.149617 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9617,"prepare.queue_time_us":89393,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":325,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":10313,"thread_start_us":1391674,"threads_started":2}
I20260924 00:22:21.165970 3423 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:21.176491 3810 consensus_meta.cc:287] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.556s user 0.013s sys 0.000s
W20260924 00:22:21.183184 2640 consensus_meta.cc:287] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf: Time spent flushing consensus metadata: real 0.697s user 0.013s sys 0.000s
I20260924 00:22:21.176756 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:21.181306 3937 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.217482 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:21.201988 3816 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:21.170090 3529 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.204021 3819 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:21.202291 3837 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:21.179440 3807 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.241168 3529 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.176558 3869 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.246214 3947 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:21.246940 3869 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.224303 3875 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:21.249905 3875 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.252002 3875 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.214421 3423 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:21.253639 3423 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.254245 3957 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:21.254736 3733 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.256242 3423 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.256757 3733 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.257576 3801 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.228634 3521 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:21.259169 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1222) took 1523 ms. Trace:
I20260924 00:22:21.259668 3521 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.211879 2640 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:21.075835 2136 rpcz_store.cc:276] 0924 00:22:19.375664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.396760 (+ 21096us) service_pool.cc:227] Handling call
0924 00:22:19.707786 (+311026us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b5800004298. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5cb7c 00007fb6d1b5c73b 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:19.707966 (+ 180us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:19.724935 (+ 16969us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:19.725356 (+ 421us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:19.749401 (+ 24045us) log.cc:842] Serialized 26 byte log entry
0924 00:22:20.869006 (+1119605us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:20.869520 (+ 514us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:20.875546 (+ 6026us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:21.025828 (+150282us) raft_consensus.cc:1671] finished
0924 00:22:21.025852 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.026095 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10340,"raft.queue_time_us":12910,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":6731,"spinlock_wait_cycles":8824704,"thread_start_us":1065364,"threads_started":1,"wal-append.queue_time_us":1274695}
I20260924 00:22:21.207554 3612 consensus_queue.cc:237] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.263572 3936 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.185084 3756 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.267060 3756 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:21.267299 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:21.214258 3810 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.268242 3700 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.254781 3957 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.269415 3971 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:21.269860 3971 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.271021 3957 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:21.272703 2143 raft_consensus.cc:2379] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:21.259519 2137 rpcz_store.cc:276] 0924 00:22:19.735586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.738109 (+ 2523us) service_pool.cc:227] Handling call
0924 00:22:19.776955 (+ 38846us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 00:22:19.805055 (+ 28100us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:19.805083 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 00:22:20.462690 (+657607us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b2c000c2d48. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d349fe11 00007fb6d34af442 00007fb6d4af7c11 00007fb6d4abc58a 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772
0924 00:22:20.463187 (+ 497us) log.cc:842] Serialized 52 byte log entry
0924 00:22:20.873074 (+409887us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:20.873162 (+ 88us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:20.873239 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:21.258896 (+385657us) raft_consensus.cc:1671] finished
0924 00:22:21.258913 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.259117 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14598,"raft.queue_time_us":27991,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":54481,"spinlock_wait_cycles":12375296,"thread_start_us":409090,"threads_started":1,"wal-append.queue_time_us":685412}
I20260924 00:22:21.327265 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:21.272424 3837 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:21.361300 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 387203us
I20260924 00:22:21.327281 3733 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:21.445658 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:21.446229 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:21.431910 3937 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:21.462929 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.271621 3819 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:21.464563 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 534960us
I20260924 00:22:21.463014 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:21.465618 2140 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:21.434242 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:21.325081 3391 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.436342 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:21.471566 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.438575 3951 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:21.481527 2140 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:21.442786 3940 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.461723 3800 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:21.512594 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.470089 3820 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:21.515961 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.480417 3807 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.451104 3733 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.497313 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.583806 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.246935 3947 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.510507 3971 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.592296 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.596719 2109 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:21.472769 3937 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.603441 3529 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:21.605029 3529 raft_consensus.cc:686] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:21.607309 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.608121 2139 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:21.584877 3810 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.618721 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:21.625164 2065 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 7
I20260924 00:22:21.625798 3391 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.628120 3700 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:21.629673 3979 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260924 00:22:21.630906 3954 log.cc:924] Time spent T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.464s user 0.000s sys 0.000s
I20260924 00:22:21.633275 2643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 311) took 2249 ms. Trace:
W20260924 00:22:21.633862 3965 log.cc:924] Time spent T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.347s user 0.000s sys 0.001s
I20260924 00:22:21.635070 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1309) took 1152 ms. Trace:
I20260924 00:22:21.634094 2643 rpcz_store.cc:276] 0924 00:22:19.384299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:19.398625 (+ 14326us) service_pool.cc:227] Handling call
0924 00:22:19.484116 (+ 85491us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:19.533634 (+ 49518us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:19.553606 (+ 19972us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:19.658421 (+104815us) log.cc:842] Serialized 26 byte log entry
0924 00:22:21.073805 (+1415384us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:21.083878 (+ 10073us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:21.089624 (+ 5746us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:21.630394 (+540770us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:21.632900 (+ 2506us) raft_consensus.cc:1671] finished
0924 00:22:21.632927 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.633186 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10466,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":49722,"spinlock_wait_cycles":14540032,"thread_start_us":1347224,"threads_started":1,"wal-append.queue_time_us":1408087}
I20260924 00:22:21.635388 2066 rpcz_store.cc:276] 0924 00:22:20.483108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.483213 (+ 105us) service_pool.cc:227] Handling call
0924 00:22:20.483496 (+ 283us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:20.483851 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:20.483890 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:20.486122 (+ 2232us) log.cc:842] Serialized 26 byte log entry
0924 00:22:20.940860 (+454738us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:20.940901 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:20.940965 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:21.634889 (+693924us) raft_consensus.cc:1671] finished
0924 00:22:21.634909 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.635022 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1896,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":454392,"threads_started":1,"wal-append.queue_time_us":707198}
I20260924 00:22:21.636322 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.637416 3924 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.637702 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.541980 3820 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:21.640595 3863 consensus_meta.cc:287] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.830s user 0.008s sys 0.005s
I20260924 00:22:21.641023 3863 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.642098 3820 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.622826 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:22:21.623728 3959 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.392s user 0.000s sys 0.001s
I20260924 00:22:21.643685 3863 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.643767 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.644487 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1365) took 1434 ms. Trace:
I20260924 00:22:21.606218 3529 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.645732 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:21.646327 2064 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:21.647497 2640 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260924 00:22:21.648427 3920 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:21.598621 2646 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:21.648833 3684 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:21.649593 3684 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.650548 2144 raft_consensus.cc:2379] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:21.651887 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1312) took 1160 ms. Trace:
I20260924 00:22:21.652256 3684 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.652222 2070 rpcz_store.cc:276] 0924 00:22:20.491959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.492344 (+ 385us) service_pool.cc:227] Handling call
0924 00:22:20.493292 (+ 948us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:20.619824 (+126532us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:21.651681 (+1031857us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54000b2554. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:21.651836 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":288725888}
I20260924 00:22:21.653940 3733 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.656975 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.624001 3976 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:21.624294 3819 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.587419 3890 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:21.587483 3968 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:21.586262 3975 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:21.658568 2035 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:21.515463 3951 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.523362 3800 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.665105 2065 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:21.644819 1987 rpcz_store.cc:276] 0924 00:22:20.210141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.210287 (+ 146us) service_pool.cc:227] Handling call
0924 00:22:20.210658 (+ 371us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:20.215326 (+ 4668us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:20.215370 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:20.222984 (+ 7614us) log.cc:842] Serialized 26 byte log entry
0924 00:22:20.723510 (+500526us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:20.723545 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:20.723609 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:21.644299 (+920690us) raft_consensus.cc:1671] finished
0924 00:22:21.644319 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:21.644438 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5543,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":1683,"thread_start_us":493865,"threads_started":1,"wal-append.queue_time_us":780732}
I20260924 00:22:21.672785 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.686220 1960 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:21.670302 3961 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:21.723016 3858 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:21.723479 3858 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:21.723879 3858 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:21.719424 3968 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.726363 3924 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.727808 3968 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.655943 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.656008 3950 raft_consensus.cc:3032] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:21.719321 3890 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.733325 3961 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.730630 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.733821 2139 raft_consensus.cc:2379] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 2: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:21.735486 3961 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.737012 3890 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.648900 3920 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:21.740653 3920 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.675750 3863 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:21.741658 3863 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.729913 3950 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:21.742015 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:21.738157 3858 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:21.746716 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 88220.5us
I20260924 00:22:21.615525 3894 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:21.730659 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:21.712234 3976 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:21.750275 3985 log.cc:924] Time spent T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260924 00:22:21.718291 3733 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260924 00:22:21.753500 3937 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.754500 3947 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.755777 3961 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:21.989293 3956 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:22.324127 3961 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.324559 3956 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:22.094547 2141 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.017s sys 0.035s
W20260924 00:22:22.077548 2613 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.000s sys 0.034s
W20260924 00:22:22.327281 2613 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.848s user 0.000s sys 0.045s
W20260924 00:22:22.095345 2642 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.012s sys 0.039s
W20260924 00:22:22.327852 2642 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s user 0.020s sys 0.042s
I20260924 00:22:22.327888 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:21.951368 3924 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.329188 3922 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.328868 3924 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.329778 3922 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:21.976003 3915 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:21.745966 3482 consensus_meta.cc:287] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.798s user 0.006s sys 0.006s
I20260924 00:22:22.332551 3482 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.333343 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 1596) took 1606 ms. Trace:
W20260924 00:22:22.256794 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.000s sys 0.024s
I20260924 00:22:22.333889 1995 rpcz_store.cc:276] 0924 00:22:20.727383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.746773 (+ 19390us) service_pool.cc:227] Handling call
0924 00:22:22.333058 (+1586285us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b5400101654. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:22.333232 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":561960192}
W20260924 00:22:22.326977 2141 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s user 0.025s sys 0.037s
W20260924 00:22:22.334828 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.854s user 0.000s sys 0.027s
W20260924 00:22:21.753475 3816 consensus_meta.cc:287] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.511s user 0.007s sys 0.003s
W20260924 00:22:22.335318 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 855174us
I20260924 00:22:21.755659 3819 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 00:22:22.235467 3521 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.006s sys 0.009s
W20260924 00:22:22.337119 3521 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.006s sys 0.013s
W20260924 00:22:22.327628 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 848024us
I20260924 00:22:22.005035 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:21.755829 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.331339 3915 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.347505 2035 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:22.348359 2071 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
W20260924 00:22:22.355191 2068 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.009s sys 0.008s
W20260924 00:22:22.358501 2068 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.643s user 0.010s sys 0.008s
I20260924 00:22:22.358754 1992 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:22.348950 2065 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:22.346876 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1385) took 1143 ms. Trace:
I20260924 00:22:22.383040 1991 rpcz_store.cc:276] 0924 00:22:21.203722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.203853 (+ 131us) service_pool.cc:227] Handling call
0924 00:22:21.204253 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:21.269590 (+ 65337us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:21.269632 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:21.269685 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:21.269735 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.346578 (+1076843us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b540011c954. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:22.346634 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.346809 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":149770624}
W20260924 00:22:21.982635 3981 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
W20260924 00:22:22.386409 1992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 1383) took 1210 ms (client timeout 1611 ms). Trace:
I20260924 00:22:22.357357 3894 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:22.247954 2646 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.002s sys 0.011s
W20260924 00:22:22.399165 2646 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s user 0.004s sys 0.011s
I20260924 00:22:22.357731 2136 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:22.387534 3733 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.400224 3733 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.400560 3894 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.388087 3929 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:22.387565 2062 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47766 (request call id 1141) took 1828 ms (client timeout 1748 ms). Trace:
I20260924 00:22:22.388818 2035 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:22.395632 3924 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.403872 2035 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:22.390682 3991 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:22.326473 3961 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:21.720715 3975 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:22.351480 3529 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.010s sys 0.006s
I20260924 00:22:22.393932 3858 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.394016 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:22:22.234970 3837 consensus_meta.cc:287] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.800s user 0.008s sys 0.002s
I20260924 00:22:22.394840 3890 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:22.395434 3801 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.409076 3924 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260924 00:22:22.396260 1992 rpcz_store.cc:269] 0924 00:22:21.176497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.176702 (+ 205us) service_pool.cc:227] Handling call
0924 00:22:21.755632 (+578930us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:22.386335 (+630703us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4392704}
W20260924 00:22:22.402180 2062 rpcz_store.cc:269] 0924 00:22:20.525005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.534945 (+ 9940us) service_pool.cc:227] Handling call
0924 00:22:22.352669 (+1817724us) spinlock_profiling.cc:243] Waited 46.6 ms on lock 0x7b5400468554. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:22.353069 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":107294464}
I20260924 00:22:21.758536 3800 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:21.920928 3702 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.144430 3982 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.357123 3863 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:22.405288 2138 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.009s sys 0.038s
W20260924 00:22:22.454128 3529 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.807s user 0.011s sys 0.007s
I20260924 00:22:22.335578 3816 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:22.488027 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 84255.5us
I20260924 00:22:22.492239 3797 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:22.499737 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 105283us
I20260924 00:22:22.482961 3858 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.482645 3991 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:22.509647 2035 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:22.468514 3993 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:22.498080 2138 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s user 0.009s sys 0.038s
I20260924 00:22:22.513267 3858 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.490314 3733 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.537626 3482 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.536899 4007 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.495095 3947 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.536432 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:22.539484 3816 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.482515 3837 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.536595 3993 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.536727 3756 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:22.542627 3996 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.506058 2640 raft_consensus.cc:2379] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:22.495808 3819 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:22.576722 3996 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.579653 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 1234) took 1484 ms. Trace:
I20260924 00:22:22.490051 3890 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.582744 3890 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:22.501863 3982 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:22.491050 3801 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.486822 4000 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:22.580466 2138 rpcz_store.cc:276] 0924 00:22:21.095635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.101759 (+ 6124us) service_pool.cc:227] Handling call
0924 00:22:21.178464 (+ 76705us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:21.181722 (+ 3258us) raft_consensus.cc:1520] Early marking committed up to index 6
0924 00:22:21.621000 (+439278us) log.cc:842] Serialized 14 byte log entry
0924 00:22:22.558671 (+937671us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:22.562495 (+ 3824us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:22:22.564585 (+ 2090us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.568501 (+ 3916us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.579540 (+ 11039us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":58962,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"thread_start_us":1235069,"threads_started":2,"wal-append.queue_time_us":877857}
I20260924 00:22:22.537079 2035 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:22.536770 3862 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.452332 3929 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.502271 3702 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.514540 3991 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.515672 3797 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:22.569609 3997 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260924 00:22:22.569999 3850 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:22.570799 3975 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:22.590500 3999 log.cc:924] Time spent T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260924 00:22:22.576233 4013 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.617308 4012 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.617147 4013 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.617877 4012 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.619349 4013 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.590734 3819 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:22.619751 4012 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:22.583169 4004 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260924 00:22:22.620329 4015 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.620818 4015 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.622433 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60184 (request call id 1621) took 1162 ms. Trace:
I20260924 00:22:22.622943 4015 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.592867 3862 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.624102 2136 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:22.625273 3862 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.584656 3982 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:22:22.608119 3993 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.584829 3801 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:22.587850 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 52646.4us
I20260924 00:22:22.591324 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.551463 3863 consensus_queue.cc:1035] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260924 00:22:22.628762 2035 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:22.566212 3733 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.629561 3733 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:22.580928 3972 log.cc:924] Time spent T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 1.381s user 0.009s sys 0.008s
I20260924 00:22:22.573647 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 1608) took 1134 ms. Trace:
I20260924 00:22:22.575625 3816 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.631567 3733 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.631899 3816 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.567039 2644 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:22.633311 2035 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:22.634055 3993 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:22.634363 3816 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.575315 3482 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.562885 4007 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.635353 3482 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.637306 4007 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.634435 3993 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:22.638753 3993 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:22.630986 2141 rpcz_store.cc:276] 0924 00:22:21.439557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.439707 (+ 150us) service_pool.cc:227] Handling call
0924 00:22:21.440133 (+ 426us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:21.462931 (+ 22798us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:21.462958 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:21.463372 (+ 414us) log.cc:842] Serialized 26 byte log entry
0924 00:22:22.335153 (+871781us) raft_consensus.cc:1631] Marking committed up to 2
0924 00:22:22.335190 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.335252 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:22.573380 (+238128us) raft_consensus.cc:1671] finished
0924 00:22:22.573405 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.573573 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":75914,"raft.queue_time_us":24009,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":26360,"thread_start_us":820127,"threads_started":1,"wal-append.queue_time_us":993938}
I20260924 00:22:22.611358 3702 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.577829 3947 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.642347 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 1615) took 1197 ms. Trace:
I20260924 00:22:22.642467 3947 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.612511 3837 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.642825 2145 rpcz_store.cc:276] 0924 00:22:21.445444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.735801 (+290357us) service_pool.cc:227] Handling call
0924 00:22:21.736085 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:22.336372 (+600287us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:22.336401 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:22.336438 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:22.336470 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.642132 (+305662us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54003309d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:22.642174 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.642294 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1380010240}
I20260924 00:22:22.637349 3482 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.613835 3797 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.614588 2142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 1611) took 1174 ms. Trace:
I20260924 00:22:22.646036 2642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 314) took 1769 ms. Trace:
I20260924 00:22:22.644960 2142 rpcz_store.cc:276] 0924 00:22:21.440466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.448495 (+ 8029us) service_pool.cc:227] Handling call
0924 00:22:21.471809 (+ 23314us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:21.492729 (+ 20920us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:21.492757 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:21.510424 (+ 17667us) log.cc:842] Serialized 26 byte log entry
0924 00:22:22.074392 (+563968us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:22.137617 (+ 63225us) log.cc:842] Serialized 14 byte log entry
0924 00:22:22.486003 (+348386us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.486109 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:22.614320 (+128211us) raft_consensus.cc:1671] finished
0924 00:22:22.614349 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.614517 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":272465,"prepare.queue_time_us":69466,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":2543,"raft.queue_time_us":277197,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":1581,"thread_start_us":592536,"threads_started":2,"wal-append.queue_time_us":941843}
I20260924 00:22:22.615047 3858 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.583847 4000 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.646467 2642 rpcz_store.cc:276] 0924 00:22:20.877056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.882437 (+ 5381us) service_pool.cc:227] Handling call
0924 00:22:20.946658 (+ 64221us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:21.080703 (+134045us) raft_consensus.cc:1520] Early marking committed up to index 7
0924 00:22:21.464547 (+383844us) log.cc:842] Serialized 14 byte log entry
0924 00:22:22.485790 (+1021243us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:22.485848 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0924 00:22:22.485896 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.645744 (+159848us) spinlock_profiling.cc:243] Waited 94.6 ms on lock 0x7b540050f854. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:22.645803 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.645950 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24245,"prepare.queue_time_us":431033,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":164306,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":13609,"spinlock_wait_cycles":217522944,"thread_start_us":1315632,"threads_started":3,"wal-append.queue_time_us":1017194}
I20260924 00:22:22.605578 4017 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.608282 3929 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.649228 4017 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.622969 2139 rpcz_store.cc:276] 0924 00:22:21.459923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.637404 (+177481us) service_pool.cc:227] Handling call
0924 00:22:22.622191 (+984787us) spinlock_profiling.cc:243] Waited 95.2 ms on lock 0x7b5400451854. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:22.622362 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":218940160}
I20260924 00:22:22.651420 4017 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.632432 3391 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:22.633040 2644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45582 (request call id 224) took 1445 ms. Trace:
I20260924 00:22:22.652908 4000 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.604790 3800 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.581002 3996 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.653755 3800 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.655720 3800 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:22.561870 3976 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:22.653426 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 1580) took 2100 ms. Trace:
I20260924 00:22:22.603942 3850 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:22.657531 3850 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.658566 3850 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:22.661415 3951 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.634783 3956 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:22.662434 2143 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57748 (request call id 1229) took 1782 ms (client timeout 2187 ms). Trace:
W20260924 00:22:22.662842 2143 rpcz_store.cc:269] 0924 00:22:20.879904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.880048 (+ 144us) service_pool.cc:227] Handling call
0924 00:22:22.662256 (+1782208us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540023d2d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:22.662379 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":788571904}
I20260924 00:22:22.591078 3971 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.653198 2644 rpcz_store.cc:276] 0924 00:22:21.187864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.192731 (+ 4867us) service_pool.cc:227] Handling call
0924 00:22:22.536210 (+1343479us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b5800005a98. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:22:22.632954 (+ 96744us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11444992}
I20260924 00:22:22.663885 3971 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.667713 4007 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.668294 4007 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:20.262046 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:22.670065 3956 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.656301 1994 rpcz_store.cc:276] 0924 00:22:20.552871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.553016 (+ 145us) service_pool.cc:227] Handling call
0924 00:22:21.599288 (+1046272us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b5800003398. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:22:21.599477 (+ 189us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:21.725408 (+125931us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:21.725442 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:21.725491 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:21.725523 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.653181 (+927658us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400299fd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:22.653231 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.653350 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":294378112}
I20260924 00:22:22.670625 3956 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.672545 3956 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.673677 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:22.673750 2648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45582 (request call id 228) took 1053 ms. Trace:
I20260924 00:22:22.673941 4007 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.674067 1890 catalog_manager.cc:5495] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:22.674378 2648 rpcz_store.cc:276] 0924 00:22:21.620455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:21.620640 (+ 185us) service_pool.cc:227] Handling call
0924 00:22:21.639072 (+ 18432us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:21.671792 (+ 32720us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:21.671827 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:21.671872 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:21.671903 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:22.673492 (+1001589us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b54002db754. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:22:22.673539 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:22.673698 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74700672}
I20260924 00:22:22.675595 1890 catalog_manager.cc:5495] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:22.679301 3800 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:22.679700 3800 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.675793 3971 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.680158 3800 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:22.634814 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:22.682912 3983 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260924 00:22:22.683878 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 49132us
I20260924 00:22:22.685509 3863 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:22.686316 3863 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.707950 3863 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.686208 3950 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.711023 3950 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:22.715936 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:22.723862 1994 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:22.739691 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.740600 1994 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:22.725591 4007 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.755110 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:22.756127 1989 raft_consensus.cc:2379] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:22.756567 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.756979 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.737344 3724 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:22.757659 1989 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:22.757848 3724 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.759578 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.760011 3724 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.760399 1987 raft_consensus.cc:2396] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260924 00:22:22.760720 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.761224 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:22.761801 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.762574 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.737406 3521 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:22.786062 1992 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:22.786566 1995 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:22.799289 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.800151 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:22.800959 2068 raft_consensus.cc:2354] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:22:22.801237 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.752585 4007 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.788103 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.789522 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.790777 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.807066 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.808039 1995 raft_consensus.cc:2379] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:22.793218 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:22.807096 1991 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:22.809667 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.794718 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:22.796444 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.796342 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.801806 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.810722 2062 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:22.793792 1994 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:22.728073 3915 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:22.786821 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 43753.2us
I20260924 00:22:22.811595 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:22.810492 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.812107 2143 raft_consensus.cc:2379] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:22.812665 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.815536 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.817624 2145 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:22.818601 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.819900 2035 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:22.737632 4019 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.801563 3996 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.844048 2613 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:22.828465 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.818171 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:22.786132 3863 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:22.804564 4000 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:22.849576 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.811190 3971 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.804376 3724 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.854899 2613 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:22.841024 1988 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:22.863610 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:22.863602 3915 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:22.815274 3956 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.819383 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.833211 3807 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:22.836494 3869 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.857076 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:22.834898 3982 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:22.867658 2063 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:22.830456 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.869402 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.842589 3991 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:22.818745 3951 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:22.855206 3929 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:22.875078 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.800978 3521 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.843099 3936 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:22.891495 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.892724 2062 raft_consensus.cc:2396] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260924 00:22:22.884531 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 78535.4us
I20260924 00:22:22.885933 3982 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.866662 4022 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:22.905946 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.848832 3819 consensus_queue.cc:1046] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260924 00:22:22.908749 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:22.911010 2063 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:22.911089 2062 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:22.909632 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.913904 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.914508 2066 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 1.
I20260924 00:22:22.851213 3996 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.890092 3942 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:22.918470 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.883383 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:22.924168 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:22.864845 3863 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.867599 3724 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.928081 2064 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:22.925680 2062 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:22.885317 3807 raft_consensus.cc:686] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:22.867298 4000 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.924738 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:22.932710 2066 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:22.865216 3971 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.929905 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:22.931059 1960 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 1.547s (SENT) [suppressed 1 similar messages]
I20260924 00:22:22.898172 3922 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.908764 3929 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:22.874445 3956 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.947060 2069 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:22.947674 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.919231 3942 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:22.938306 2613 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:22.883492 3869 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.943516 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.891672 3993 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.964193 3992 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:22.965050 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:22.965755 4010 consensus_queue.cc:1046] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:22.966634 4020 raft_consensus.cc:3032] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:22.967384 4020 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:22.967128 2110 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:22.908797 3936 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.904071 3991 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.953336 4000 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.953805 3996 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.951014 3942 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:22.954545 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.954355 3807 consensus_queue.cc:237] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.955520 3982 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:22.955708 4024 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260924 00:22:22.956454 3956 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.995952 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:22.996629 1996 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:22.957036 3724 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:22.957167 3929 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.958161 3521 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:22.934634 3800 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:22.963173 1993 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:22.969933 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.950912 1890 catalog_manager.cc:5495] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 3, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 3 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:22.972733 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.899530 3951 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:22.854420 4019 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:22.963166 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.979915 2066 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:22.981775 2064 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:22.982522 2070 raft_consensus.cc:2354] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:22:22.982669 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:22.931456 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:22.985195 2110 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:22.958196 3971 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:22.991734 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:22.992008 3869 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:22.956764 4007 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.012076 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:22.985683 4010 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:22.968554 4020 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.015938 2069 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:23.015576 3993 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:22.994421 2644 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260924 00:22:23.035517 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 54204.1us
I20260924 00:22:23.035431 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.036075 2067 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.043170 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.044572 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:23.041846 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.053820 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.111409 2069 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:23.016609 2613 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:23.054577 2036 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:23.050138 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.062084 2064 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:23.051823 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:23.056407 4030 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
W20260924 00:22:23.117697 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 87999.1us
W20260924 00:22:23.119297 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 188571us
I20260924 00:22:23.092934 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.044822 3976 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:23.051326 4010 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:23.121656 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 62623.1us
I20260924 00:22:23.121836 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:23.122292 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.123957 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:23.126379 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 109886us
I20260924 00:22:23.127825 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.141620 2068 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:23.144308 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.058616 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:23.145295 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.149176 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.151523 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.152196 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:23.152726 2068 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 6
I20260924 00:22:23.152642 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.153352 2066 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:23.154454 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.155278 2069 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:23.156284 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.151541 2110 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.126200 3988 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:23.126962 4020 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:23.173334 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.181401 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.182541 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:23.178952 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 161876us
I20260924 00:22:23.178838 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.138237 4006 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:23.121311 1988 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:22:23.125335 3976 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.123082 4007 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:23.145457 4010 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
W20260924 00:22:23.176129 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 55999us
I20260924 00:22:23.185495 2068 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:23.178586 2110 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.198320 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.181387 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.182782 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.201455 3951 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.177479 3988 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:23.205426 3988 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:23.206856 4019 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.207551 3936 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.207995 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:23.188395 3521 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:23.209465 3521 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.186724 4012 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20260924 00:22:23.209889 3521 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:23.159085 3863 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.213927 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.214293 2063 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:23.214775 1996 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:23.214011 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.215154 2066 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:23.216588 2063 raft_consensus.cc:3032] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:23.217829 3529 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:23.217936 2063 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:23.218536 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.219170 1996 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:23.217433 2110 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:23.218974 2063 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:23.225495 2062 raft_consensus.cc:2379] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:23.228144 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.228672 2614 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:23.227588 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.228922 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.230471 2063 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:23.237866 1987 raft_consensus.cc:2379] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:23.192384 2064 raft_consensus.cc:2396] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260924 00:22:23.231681 2110 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.225003 3879 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.227345 4014 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:23.240417 4006 raft_consensus.cc:2781] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:22:23.240865 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.240551 3879 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:23.242264 4037 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260924 00:22:23.196972 3971 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.240816 4006 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.243019 3971 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.243340 4006 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:23.217854 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.243254 3879 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.199652 3976 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:23.244957 3971 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.258622 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.221956 3529 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.261771 2110 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.235596 3991 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:23.263142 1992 raft_consensus.cc:2354] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:23.289196 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.289412 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.289623 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.290429 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.290812 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.291352 2138 raft_consensus.cc:2379] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.291662 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.291368 1995 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:23.290215 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.293962 2145 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:23.295157 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.297163 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.297690 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.298038 2145 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:23.298723 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.299634 2141 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.308432 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.310180 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.290288 1996 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:23.260681 3800 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:23.264031 3529 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:23.319355 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.323021 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.328220 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:23.329314 4032 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:23.329856 4032 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:23.330261 4032 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:23.331236 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.332110 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.332927 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.327126 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.334079 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.334373 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.335108 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.335456 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.336130 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.336318 2141 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:23.337702 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.338119 2110 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.339411 3800 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:23.339862 3800 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.340291 3800 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:23.200132 4007 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.281800 3801 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.355513 2110 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.356325 4009 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:23.356676 2110 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:23.356683 4009 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.357664 4009 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.357710 3975 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:23.374536 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 233417us
I20260924 00:22:23.374943 2145 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:23.258630 4014 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260924 00:22:23.384509 4014 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:23.384879 4014 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:23.385733 4014 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.387899 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:23.392441 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:23.396471 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.407115 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.410336 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:23.363888 3801 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:23.391297 3920 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260924 00:22:23.395504 3915 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:23.407270 3996 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:23.401280 3940 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:23.399580 3976 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.464900 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.474131 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.474712 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.490819 2140 raft_consensus.cc:2354] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:23.491060 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.480739 2137 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:23.491570 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.411751 4014 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.453035 4008 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:23.495155 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:23.497573 2145 raft_consensus.cc:595] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.497980 2145 raft_consensus.cc:603] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:23.505121 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.506793 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.510370 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.507498 2137 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:23.510849 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.510970 1989 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:23.511680 2142 raft_consensus.cc:2354] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:23.512439 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.483726 3940 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:23.496690 4008 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:23.543735 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:23.551038 4042 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
I20260924 00:22:23.553329 4008 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.523180 3940 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:23.557543 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:23.558686 4043 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260924 00:22:23.509627 3800 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.563191 4007 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.564437 3800 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:23.565639 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 61672.4us
I20260924 00:22:23.566154 2141 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:23.558777 1989 raft_consensus.cc:2396] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260924 00:22:23.562387 4010 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.572154 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.575103 1995 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:23.579288 2137 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:23.483878 3915 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:23.583801 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:23.589751 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 187416us
W20260924 00:22:23.573786 4052 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.113s user 0.010s sys 0.006s
I20260924 00:22:23.576639 3976 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.591020 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.591617 2143 raft_consensus.cc:2396] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260924 00:22:23.591089 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.595829 1989 raft_consensus.cc:2354] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:23.596869 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.598569 1995 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:23.599750 2110 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.604398 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.606958 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.607014 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.609397 2613 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.613212 3915 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.615499 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.616604 2145 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.618276 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.618536 2613 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.626206 3940 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:23.628211 3940 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.629779 3988 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.630501 4010 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.630710 2067 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:22:23.631711 3988 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.635113 2109 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:23.636523 4006 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:23.637727 3992 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:23.638723 4006 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:23.641336 3942 consensus_meta.cc:287] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.646s user 0.009s sys 0.001s
I20260924 00:22:23.641733 3942 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.646569 4009 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.647786 3942 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.648875 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.620497 3937 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:23.654469 2142 raft_consensus.cc:1226] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:23.652479 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:23.656191 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.657605 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:23.661953 4038 log.cc:924] Time spent T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.481s user 0.000s sys 0.003s
I20260924 00:22:23.672117 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.620611 2641 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260924 00:22:23.649122 3801 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:23.675004 4046 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.417s user 0.003s sys 0.000s
I20260924 00:22:23.662428 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.598158 3521 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.677110 2145 raft_consensus.cc:1261] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:23.667799 4008 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.678192 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.710654 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.670967 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.660566 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:23.719673 2143 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.665656 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.668905 3975 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:23.660416 3529 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:23.658535 4048 log.cc:924] Time spent T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.277s user 0.002s sys 0.000s
I20260924 00:22:23.657944 1990 raft_consensus.cc:2354] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:23.749985 2137 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
W20260924 00:22:23.756022 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 99281.7us
I20260924 00:22:23.763723 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33246 (request call id 216) took 5605 ms. Trace:
I20260924 00:22:23.602557 4020 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:23.658607 3879 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:23.698544 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.723696 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.723984 3937 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260924 00:22:23.724463 4055 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260924 00:22:23.766644 2139 raft_consensus.cc:2379] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.833767 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.839102 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.840960 2613 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.842350 2142 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:23.843155 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.843739 1995 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.834339 2137 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:23.766413 1891 rpcz_store.cc:276] 0924 00:22:18.157883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:18.158344 (+ 461us) service_pool.cc:227] Handling call
0924 00:22:23.763651 (+5605307us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:19.464260 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:19.464927 (+ 667us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:19.464972 (+ 45us) write_op.cc:435] Acquired schema lock
0924 00:22:19.465002 (+ 30us) tablet.cc:660] Decoding operations
0924 00:22:19.474684 (+ 9682us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:19.474773 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:19.474801 (+ 28us) tablet.cc:683] Acquiring locks for 9 operations
0924 00:22:19.478332 (+ 3531us) tablet.cc:699] Row locks acquired
0924 00:22:19.478357 (+ 25us) write_op.cc:260] PREPARE: finished
0924 00:22:19.478505 (+ 148us) write_op.cc:270] Start()
0924 00:22:19.478622 (+ 117us) write_op.cc:276] Timestamp: P: 1790209339478459 usec, L: 0
0924 00:22:19.478640 (+ 18us) op_driver.cc:331] REPLICATION: starting
0924 00:22:19.479284 (+ 644us) log.cc:842] Serialized 3587 byte log entry
0924 00:22:19.525263 (+ 45979us) op_driver.cc:444] REPLICATION: finished
0924 00:22:19.590619 (+ 65356us) write_op.cc:301] APPLY: starting
0924 00:22:19.611843 (+ 21224us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:19.681283 (+ 69440us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:19.686338 (+ 5055us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:19.937050 (+250712us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:20.005854 (+ 68804us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0924 00:22:20.008662 (+ 2808us) write_op.cc:312] APPLY: finished
0924 00:22:20.123378 (+114716us) log.cc:842] Serialized 71 byte log entry
0924 00:22:20.140254 (+ 16876us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:20.154097 (+ 13843us) write_op.cc:454] Released schema lock
0924 00:22:20.156263 (+ 2166us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:20.209863 (+ 53600us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:20.211711 (+ 1848us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5029,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":27437,"apply.run_cpu_time_us":13606,"apply.run_wall_time_us":628110,"num_ops":9,"prepare.queue_time_us":25118,"prepare.run_cpu_time_us":4401,"prepare.run_wall_time_us":33236,"replication_time_us":32941,"thread_start_us":27128,"threads_started":1}]]}
I20260924 00:22:23.848233 2137 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:23.765738 4008 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.855073 1995 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:23.855157 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.857448 2110 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:23.858402 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.860074 3980 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:23.799116 2138 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.864936 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.868240 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:23.849803 3872 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:23.876892 3529 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.845893 4020 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:23.882887 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.883742 2065 raft_consensus.cc:2354] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:22:23.883406 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:23.884748 2035 leader_election.cc:398] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:23.885025 2035 leader_election.cc:401] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260924 00:22:23.887033 2644 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:23.887367 3928 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:23.888264 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:23.888701 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:23.889530 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.892620 2143 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
W20260924 00:22:23.895382 4032 consensus_meta.cc:287] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.565s user 0.011s sys 0.000s
I20260924 00:22:23.895982 4032 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.897423 2144 raft_consensus.cc:2379] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.897450 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:23.901798 3521 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.902267 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.902741 4008 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:23.903414 4008 raft_consensus.cc:686] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:23.904608 4008 consensus_queue.cc:237] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.907712 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.907759 2035 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:23.911028 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.914043 2140 raft_consensus.cc:1226] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:23.883829 4020 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:23.916733 2035 leader_election.cc:398] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:23.917039 2035 leader_election.cc:401] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:23.914476 3928 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:23.919932 3928 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:23.919057 3991 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:23.920349 3928 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.919236 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:23.921434 3991 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260924 00:22:23.924311 1942 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 8c129c01e1c047628f6a87b001aa7258 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:23.924996 3991 consensus_queue.cc:1046] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:23.928381 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.930968 1993 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
W20260924 00:22:23.927923 4061 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260924 00:22:23.928795 4009 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:23.936264 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 49422.4us
I20260924 00:22:23.936954 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:23.937740 2109 leader_election.cc:398] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:22:23.934659 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.942056 2109 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.942828 1993 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:23.943678 3521 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:23.947672 2109 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.949368 1992 raft_consensus.cc:1261] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:23.951045 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.949267 4014 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:23.953053 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.953473 4032 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:23.956233 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:23.954896 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.957295 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.959111 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.959335 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.959873 2144 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:23.959410 4050 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:23.960253 1990 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.938807 3983 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260924 00:22:23.932350 3807 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260924 00:22:23.963523 3807 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:23.963848 3807 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:23.964665 3807 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.934422 3991 consensus_queue.cc:1046] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260924 00:22:23.966188 3933 raft_consensus.cc:3032] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:23.966534 3933 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:23.967504 3933 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.968737 3933 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:23.966523 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.952637 3928 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.972965 3807 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.960090 3982 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:23.976553 3982 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:23.977373 3982 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:23.979485 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:23.981976 2109 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:23.984685 4014 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:23.985921 2066 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:23.979779 3928 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:23.988688 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:23.989228 3982 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:23.994730 2109 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:23.995649 1990 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:23.998167 4028 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:23.998834 4028 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:23.999303 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:23.999879 4028 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.009433 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.010944 4020 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.012804 1991 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 2.
I20260924 00:22:24.013466 4014 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:24.014595 2142 raft_consensus.cc:1226] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:24.015224 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:24.014565 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.016492 4014 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:24.016646 4050 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:24.016957 4014 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:24.017735 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:24.022300 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 13ff368f645e4ee7acdef88492f25ae7, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:24.022753 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 13ff368f645e4ee7acdef88492f25ae7, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.026367 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:24.026242 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:24.028117 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:24.031936 4020 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:24.035017 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:24.040982 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:24.034427 4014 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.042485 2136 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
W20260924 00:22:24.044138 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 771618125e684526a907a103e7694ae8, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:24.044555 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 771618125e684526a907a103e7694ae8, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.045179 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:24.046002 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:24.048231 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.048790 2066 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.049960 4014 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:24.044442 2109 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.074083 4014 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:24.074613 4014 raft_consensus.cc:686] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:24.075489 4014 consensus_queue.cc:237] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.056124 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 130) took 3592 ms. Trace:
W20260924 00:22:24.056384 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 59464.4us
I20260924 00:22:24.079094 2109 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.082216 4050 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:24.082931 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.081225 1890 rpcz_store.cc:276] 0924 00:22:20.463509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:20.463653 (+ 144us) service_pool.cc:227] Handling call
0924 00:22:24.056054 (+3592401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:23.645527 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:23.646023 (+ 496us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:23.646058 (+ 35us) write_op.cc:435] Acquired schema lock
0924 00:22:23.646072 (+ 14us) tablet.cc:660] Decoding operations
0924 00:22:23.646629 (+ 557us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:23.646680 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:23.646714 (+ 34us) tablet.cc:683] Acquiring locks for 3 operations
0924 00:22:23.647042 (+ 328us) tablet.cc:699] Row locks acquired
0924 00:22:23.647057 (+ 15us) write_op.cc:260] PREPARE: finished
0924 00:22:23.647198 (+ 141us) write_op.cc:270] Start()
0924 00:22:23.647302 (+ 104us) write_op.cc:276] Timestamp: P: 1790209343647156 usec, L: 0
0924 00:22:23.647317 (+ 15us) op_driver.cc:331] REPLICATION: starting
0924 00:22:23.647904 (+ 587us) log.cc:842] Serialized 1284 byte log entry
0924 00:22:24.012039 (+364135us) op_driver.cc:444] REPLICATION: finished
0924 00:22:24.040133 (+ 28094us) write_op.cc:301] APPLY: starting
0924 00:22:24.040224 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:24.040345 (+ 121us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:24.040476 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:24.045624 (+ 5148us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:24.048652 (+ 3028us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0924 00:22:24.048697 (+ 45us) write_op.cc:312] APPLY: finished
0924 00:22:24.048909 (+ 212us) log.cc:842] Serialized 35 byte log entry
0924 00:22:24.050414 (+ 1505us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:24.050511 (+ 97us) write_op.cc:454] Released schema lock
0924 00:22:24.050559 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:24.050815 (+ 256us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:24.050839 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1333094,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":27417,"apply.run_cpu_time_us":2115,"apply.run_wall_time_us":10965,"num_ops":3,"prepare.queue_time_us":202836,"prepare.run_cpu_time_us":18461,"prepare.run_wall_time_us":212171,"raft.queue_time_us":104538,"raft.run_cpu_time_us":3203,"raft.run_wall_time_us":26582,"replication_time_us":364628,"thread_start_us":351084,"threads_started":4,"wal-append.queue_time_us":225148}]]}
I20260924 00:22:24.087113 3982 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:24.082567 2109 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.090310 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:24.093493 2140 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.093971 4014 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:24.094005 2036 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.099565 2036 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.100204 3933 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:24.100931 3807 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:24.102133 2036 leader_election.cc:398] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:24.103422 2067 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:24.103761 2066 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:24.105473 2036 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.106313 3982 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:24.106629 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.107339 3983 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:24.107712 3983 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:24.100776 2140 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:24.108098 3983 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:24.110337 2036 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.111090 3807 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:24.111478 3807 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:24.112097 3807 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.111459 2036 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.115278 2036 leader_election.cc:398] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:24.117425 2036 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.117586 3928 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:24.119920 3993 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:24.128731 2036 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.130304 3983 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.138080 3988 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:24.148542 4014 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:24.149325 3982 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:24.149080 4014 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.149878 3982 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.151898 3982 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.153263 4014 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:24.153847 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:24.154165 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 57811.6us
I20260924 00:22:24.154460 1991 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:24.155611 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:24.153887 3983 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.157171 4020 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:24.157557 4020 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:24.157713 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.158116 4020 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:24.162515 4008 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:24.162792 2036 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.163563 2646 raft_consensus.cc:1226] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 1. Current term is 2. Ops: []
I20260924 00:22:24.163892 2036 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.164565 3982 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:24.165117 3982 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:24.165705 3982 consensus_queue.cc:1046] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:24.166208 2109 leader_election.cc:398] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:22:24.166714 3982 raft_consensus.cc:3032] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Stepping down as leader of term 1
I20260924 00:22:24.167119 2109 leader_election.cc:401] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:24.167074 3982 raft_consensus.cc:729] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:24.167781 3807 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:24.168397 3982 consensus_queue.cc:260] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:24.169705 3982 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:22:24.170352 3807 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:22:24.171115 4020 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.173672 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:24.174188 1991 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:24.066849 1892 catalog_manager.cc:5495] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 2, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 2 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:24.178563 4020 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:24.180123 1991 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:24.179442 1892 catalog_manager.cc:5495] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 3 to 4, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 4 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:24.181046 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:24.181788 2109 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.181957 4014 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:22:24.182643 4014 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:24.183089 4008 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:24.183429 2109 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.184232 2036 leader_election.cc:398] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:24.184176 4014 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:24.185550 4014 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.184758 2036 leader_election.cc:401] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:24.189570 3975 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:24.196424 2036 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:24.191823 2109 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.197306 3807 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:24.197381 3975 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:24.199182 3988 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:24.199179 2138 raft_consensus.cc:1226] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:24.200094 3983 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:24.201025 3928 consensus_queue.cc:1035] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:24.205432 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:24.205966 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:24.206348 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:24.206630 2138 raft_consensus.cc:1261] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:24.206761 2139 raft_consensus.cc:2396] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260924 00:22:24.207141 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 45157us
I20260924 00:22:24.206516 2136 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:24.208333 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:24.209733 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.210512 3942 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:24.210659 3982 consensus_queue.cc:1046] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:24.210945 3942 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:24.211526 3942 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.211935 3988 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:24.212432 3988 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:24.213299 3988 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.214433 3988 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.217197 2035 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.219906 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.220516 3928 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:24.220389 3942 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:24.220939 3928 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:24.221449 3928 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:24.222016 3993 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:24.222678 3993 consensus_queue.cc:1046] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:24.223724 3933 raft_consensus.cc:3032] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:24.224073 3933 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:24.224849 3933 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.226022 3933 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.226493 2646 raft_consensus.cc:1226] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:24.229492 3991 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:24.233543 2035 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.234460 3993 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:24.234956 3993 raft_consensus.cc:686] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:24.235824 3993 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.238369 3991 consensus_queue.cc:1046] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:24.239535 3991 consensus_queue.cc:1046] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:24.240783 3991 raft_consensus.cc:3032] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:24.241248 3991 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:24.242250 3991 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.243501 3982 consensus_queue.cc:1046] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:24.244807 3991 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.245211 3982 raft_consensus.cc:3032] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:24.243341 3807 raft_consensus.cc:3032] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:24.246860 3982 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:24.231578 3928 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.217502 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.251362 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.252858 2067 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:24.252951 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.250291 3982 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.254994 3982 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.258232 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.259864 2067 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:24.252004 3928 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.251942 3933 consensus_queue.cc:1035] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260924 00:22:24.265030 2065 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:24.247001 3807 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
W20260924 00:22:24.268013 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 50597.6us
I20260924 00:22:24.291572 1960 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.266498 3807 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:24.269552 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:24.271353 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:24.293149 2140 raft_consensus.cc:595] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.293495 2140 raft_consensus.cc:603] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:24.293779 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.293567 3807 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:24.272676 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.267382 3929 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:24.298939 3929 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:24.299357 3929 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.273661 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.279163 3975 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:24.280591 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.322912 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:24.283795 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:24.289070 2062 pending_rounds.cc:76] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 5
I20260924 00:22:24.267699 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.328173 2067 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:24.267912 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:24.337694 1890 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 4 to 5, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 5 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:24.339691 1890 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 3 to 4, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:24.327644 2062 pending_rounds.cc:97] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260924 00:22:24.343075 2062 raft_consensus.cc:2866] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:24.344411 1991 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:24.267187 3942 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:24.350860 2067 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:24.341002 1890 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 4, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:24.356021 2109 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.308151 2063 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:24.335402 4032 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:24.358269 4050 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260924 00:22:24.360636 2065 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
W20260924 00:22:24.417328 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 125813us
W20260924 00:22:24.418402 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 59504.2us
I20260924 00:22:24.420578 2140 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260924 00:22:24.422119 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.424156 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.427537 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.338285 4020 consensus_queue.cc:1046] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260924 00:22:24.361824 4028 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:24.469470 4028 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:24.470425 4028 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.362942 3942 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:24.449404 2066 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:24.460378 4009 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:24.461534 4020 raft_consensus.cc:3032] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:24.466958 2109 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.422857 3521 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:24.423228 4042 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260924 00:22:24.485042 2109 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.488016 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.494028 2065 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:24.495174 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:24.495982 2143 raft_consensus.cc:595] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:24.480389 4020 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:24.503028 2109 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.490309 4009 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:24.492722 3529 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:24.496954 3975 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:24.510318 1991 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260924 00:22:24.520170 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 96113.8us
I20260924 00:22:24.486161 3521 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.521271 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33238 (request call id 229) took 7500 ms. Trace:
I20260924 00:22:24.522667 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.501940 3993 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.016s
I20260924 00:22:24.532492 4009 raft_consensus.cc:686] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:24.533670 4009 consensus_queue.cc:237] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.505954 4008 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20260924 00:22:24.497643 3929 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:24.517251 4032 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:24.538590 1991 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:24.517223 4082 raft_consensus.cc:2741] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260924 00:22:24.522403 1892 rpcz_store.cc:276] 0924 00:22:17.021094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:17.024574 (+ 3480us) service_pool.cc:227] Handling call
0924 00:22:24.304666 (+7280092us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:24.304719 (+ 53us) write_op.cc:454] Released schema lock
0924 00:22:24.521204 (+216485us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:24.274323 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:24.274849 (+ 526us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:24.274905 (+ 56us) write_op.cc:435] Acquired schema lock
0924 00:22:24.274929 (+ 24us) tablet.cc:660] Decoding operations
0924 00:22:24.275318 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:24.275377 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:24.275407 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0924 00:22:24.275737 (+ 330us) tablet.cc:699] Row locks acquired
0924 00:22:24.275756 (+ 19us) write_op.cc:260] PREPARE: finished
0924 00:22:24.275886 (+ 130us) write_op.cc:270] Start()
0924 00:22:24.275991 (+ 105us) write_op.cc:276] Timestamp: P: 1790209344275842 usec, L: 0
0924 00:22:24.276013 (+ 22us) op_driver.cc:331] REPLICATION: starting
0924 00:22:24.276604 (+ 591us) log.cc:842] Serialized 917 byte log entry
0924 00:22:24.283062 (+ 6458us) op_driver.cc:444] REPLICATION: finished
0924 00:22:24.283350 (+ 288us) write_op.cc:301] APPLY: starting
0924 00:22:24.283419 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:24.283522 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:24.283605 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:24.283867 (+ 262us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:24.284007 (+ 140us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 00:22:24.284047 (+ 40us) write_op.cc:312] APPLY: finished
0924 00:22:24.284234 (+ 187us) log.cc:842] Serialized 29 byte log entry
0924 00:22:24.284395 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:24.284505 (+ 110us) write_op.cc:454] Released schema lock
0924 00:22:24.284543 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":6080,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":1584,"apply.run_wall_time_us":1583,"num_ops":2,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":2527,"prepare.run_wall_time_us":2526,"replication_time_us":6951}]]}
I20260924 00:22:24.533613 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.538086 3993 consensus_queue.cc:1225] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer 8c129c01e1c047628f6a87b001aa7258 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260924 00:22:24.572177 4082 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:24.599048 4082 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:24.599546 4082 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:24.600013 4088 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:24.600492 4088 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:24.600841 4088 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:24.569375 4032 raft_consensus.cc:686] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:22:24.611723 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 106803us
I20260924 00:22:24.536345 4020 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:24.601583 4093 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:24.634152 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:24.634989 2140 raft_consensus.cc:595] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:24.642707 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 122776us
W20260924 00:22:24.642937 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 69706.1us
I20260924 00:22:24.619781 3942 raft_consensus.cc:686] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:24.631290 4093 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:24.657232 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:24.633008 3872 consensus_queue.cc:1046] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260924 00:22:24.674005 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:24.676124 2140 raft_consensus.cc:595] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.653461 4020 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:24.633309 4008 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260924 00:22:24.678876 3942 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.655434 4093 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:24.687435 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:24.688112 2066 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:24.672842 3529 raft_consensus.cc:686] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:24.692139 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:22:24.684819 3521 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:24.690946 3929 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:24.691434 3975 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:24.660952 4032 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.723615 4101 raft_consensus.cc:3032] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:24.765693 3529 consensus_queue.cc:237] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.772670 4101 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:24.773582 4101 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:24.774989 4101 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.779598 3807 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:24.780138 3807 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:24.781159 3807 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.782445 3807 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
W20260924 00:22:24.783128 4095 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20260924 00:22:24.784124 4096 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260924 00:22:24.738282 3988 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:24.803470 3975 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.758915 3991 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:24.845201 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 135939us
I20260924 00:22:24.843115 3801 raft_consensus.cc:979] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:24.822374 4108 raft_consensus.cc:979] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
W20260924 00:22:24.856156 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 322591us
I20260924 00:22:24.856946 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:24.860872 2068 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:24.863922 2071 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:24.872164 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:24.823186 3988 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.855235 4091 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:24.843561 3991 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.875854 3929 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:24.896188 4093 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.896646 3929 raft_consensus.cc:686] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:24.887377 4094 raft_consensus.cc:1067] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:24.908191 3929 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.908738 4093 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.894678 2138 raft_consensus.cc:1203] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Deduplicated request from leader. Original: 5.3->[6.4-7.5] Dedup: 6.4->[7.5-7.5]
I20260924 00:22:24.911469 2138 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: [6.4-7.5]
I20260924 00:22:24.885434 4091 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.901084 2143 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:24.901672 4108 raft_consensus.cc:1067] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:24.914461 1993 raft_consensus.cc:1226] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:24.905094 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:24.929296 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:24.930240 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260924 00:22:24.931407 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 72566.5us
I20260924 00:22:24.939191 4112 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:24.941368 4112 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:24.941682 4112 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.942871 4088 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.879570 3942 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:24.946189 3991 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.944959 4088 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.940686 1990 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:24.938715 3940 consensus_queue.cc:1046] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260924 00:22:24.960322 4115 raft_consensus.cc:3032] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:24.960759 4115 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:24.961501 4082 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.961704 4115 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:24.962607 1991 raft_consensus.cc:1226] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:24.963605 4082 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.964659 4115 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.966658 3988 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:24.966795 4091 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:24.970175 1991 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:24.972183 1988 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:24.978674 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:24.975500 1995 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:24.982908 1995 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:24.983363 1995 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:24.984508 1991 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:24.975744 1992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:24.986106 1992 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:24.986505 1992 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:24.997049 1995 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:24.985209 4088 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:25.005659 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 128856us
I20260924 00:22:25.006553 1960 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:25.007251 1995 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.023013 4112 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.018868 1891 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 7 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:25.029706 1891 catalog_manager.cc:5495] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:25.010777 4082 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:25.005045 4100 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260924 00:22:25.006394 4099 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260924 00:22:25.044566 4112 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:25.009065 4088 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:25.046972 4088 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:25.037281 1891 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 2 to 5, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 5 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:25.040187 4082 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:25.048504 1891 catalog_manager.cc:5495] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 2 to 4, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.061491 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:25.062216 2069 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:25.051636 4123 consensus_queue.cc:1046] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260924 00:22:25.064610 4123 raft_consensus.cc:3032] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:25.064921 4123 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:25.065660 4123 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.066817 4123 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:25.069677 3807 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:25.071727 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:25.072368 2071 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:25.073791 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:25.074036 2065 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:25.077724 1890 catalog_manager.cc:5495] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 3 to 5, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:25.063051 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:25.084278 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:25.085165 2143 raft_consensus.cc:2379] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:25.052212 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:25.054478 4088 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:25.059873 1891 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 4 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.105798 1891 catalog_manager.cc:5495] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
W20260924 00:22:25.108606 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 56470us
I20260924 00:22:25.107414 1891 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 4 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.111223 2109 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:25.111997 1891 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:25.113893 1891 catalog_manager.cc:5495] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.114363 4093 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:25.117851 4093 raft_consensus.cc:686] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:25.117915 2071 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:25.118999 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.119728 1891 catalog_manager.cc:5495] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 4 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.060050 4082 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:25.125346 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.126380 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.127353 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.121724 1891 catalog_manager.cc:5495] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:25.128726 1960 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:25.129292 4122 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:25.129977 4093 consensus_queue.cc:237] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.132282 3942 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:25.132678 3942 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:25.133014 3942 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:25.133612 4088 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:25.135946 4088 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:25.137084 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.137326 4020 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:25.138747 4020 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:25.138824 4008 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:25.139175 4020 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:25.140398 4028 consensus_queue.cc:1046] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:25.142304 4098 raft_consensus.cc:3032] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:25.142666 4098 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:25.143360 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:25.143509 4014 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:25.146515 4103 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:25.149539 4088 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:25.151258 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.151567 4020 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.156219 4098 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.157619 4088 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:25.157816 2065 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260924 00:22:25.158231 2065 pending_rounds.cc:76] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 4
I20260924 00:22:25.158429 2065 pending_rounds.cc:97] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260924 00:22:25.158634 2065 raft_consensus.cc:2866] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:25.161470 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.162427 2071 raft_consensus.cc:1226] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:25.162627 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.154585 1891 catalog_manager.cc:5495] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.164247 1891 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.165295 1891 catalog_manager.cc:5495] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 1 to 2, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 2 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.166082 1891 catalog_manager.cc:5495] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 6, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 6 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:25.163235 4122 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:25.169999 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.195748 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:25.128964 3929 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:25.200723 4090 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:25.167618 4112 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:25.211949 4115 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:25.212984 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 56276.2us
I20260924 00:22:25.219959 1991 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:25.218950 1992 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.222721 1990 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:25.225596 1992 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.227255 1993 raft_consensus.cc:1226] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:25.161546 4008 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20260924 00:22:25.230080 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:25.205636 4122 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:25.246631 1996 raft_consensus.cc:1226] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:25.212581 4115 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:25.248806 4115 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.251333 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:25.243443 4008 consensus_queue.cc:1225] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260924 00:22:25.252607 2143 raft_consensus.cc:2396] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260924 00:22:25.253572 4098 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:25.264688 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:25.207968 4088 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:25.265596 1993 raft_consensus.cc:2396] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260924 00:22:25.293282 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 48801.7us
I20260924 00:22:25.266577 4020 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:25.295560 2614 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:25.297915 4101 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 00:22:25.301199 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 71625.3us
I20260924 00:22:25.307185 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:25.307976 2071 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
W20260924 00:22:25.321913 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 126249us
I20260924 00:22:25.322794 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.324385 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.330196 2071 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:25.417606 1960 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.317838 4107 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260924 00:22:25.420895 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.430023 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:25.330410 4123 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:25.331252 4122 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260924 00:22:25.437505 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 118030us
I20260924 00:22:25.320782 4097 consensus_queue.cc:1046] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260924 00:22:25.420900 4090 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:25.427055 4112 raft_consensus.cc:686] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:25.330590 4020 consensus_queue.cc:1046] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20260924 00:22:25.435163 4123 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:25.547559 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 85222.8us
I20260924 00:22:25.523033 3807 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:25.527386 4110 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260924 00:22:25.558446 4136 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20260924 00:22:25.558923 4008 raft_consensus.cc:3032] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:25.533901 4123 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:25.559342 4008 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:25.527395 3940 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:25.440502 4082 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.550302 4122 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:25.519981 4108 consensus_queue.cc:1046] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260924 00:22:25.525426 4099 consensus_queue.cc:1046] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260924 00:22:25.570336 2109 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:25.574591 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 52245.4us
I20260924 00:22:25.575784 1989 raft_consensus.cc:1261] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:25.560206 4008 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.565058 4139 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:25.553947 3807 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.561264 4050 consensus_queue.cc:1046] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260924 00:22:25.567647 3940 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:25.689556 3991 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:25.622030 4107 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
W20260924 00:22:25.752329 4133 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.380s user 0.001s sys 0.000s
I20260924 00:22:25.565826 4112 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:25.769404 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 351278us
W20260924 00:22:25.770735 3942 consensus_meta.cc:287] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.637s user 0.008s sys 0.001s
I20260924 00:22:25.766981 4050 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:25.766268 3991 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.804566 4115 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:25.805357 4115 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:25.806398 3991 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.787815 4107 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:25.807938 4107 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.808005 4115 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:25.811455 4020 raft_consensus.cc:3032] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:25.812011 4020 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:25.813355 3940 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.818010 4014 raft_consensus.cc:3032] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
W20260924 00:22:25.818714 4141 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.293s user 0.001s sys 0.000s
W20260924 00:22:25.819880 4135 log.cc:924] Time spent T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.286s user 0.000s sys 0.001s
W20260924 00:22:25.821401 4134 log.cc:924] Time spent T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.268s user 0.001s sys 0.000s
I20260924 00:22:25.818115 3807 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.818638 4014 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:25.793874 3988 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:25.793030 4145 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:25.790241 3942 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.791706 4050 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.881455 4091 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:25.882402 4147 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:25.882098 4091 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:25.882941 4147 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:25.884683 4091 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:25.880693 4103 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:25.885443 4103 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.887688 4103 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:25.888262 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:25.885054 4147 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:25.889335 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:25.890012 2644 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:25.891360 2109 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:25.892578 4147 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:25.893118 4147 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:25.893566 4147 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:25.893698 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:25.894680 2138 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 6: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
W20260924 00:22:25.900209 4062 log.cc:924] Time spent T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Append to log took a long time: real 0.136s user 0.002s sys 0.000s
I20260924 00:22:25.906347 3942 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.908383 4123 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.910760 4123 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.812853 4020 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:25.840806 4145 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:25.836403 3988 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:25.912891 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 139162us
I20260924 00:22:25.840693 4082 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:25.912372 4050 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:25.912385 4014 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:26.045452 4050 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.046319 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.049973 1987 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 1.
I20260924 00:22:26.047425 1995 pending_rounds.cc:76] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 1
I20260924 00:22:26.051113 1995 pending_rounds.cc:97] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260924 00:22:26.051466 1995 raft_consensus.cc:2866] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:26.049894 3940 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:26.053659 3940 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.057474 4154 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.058002 4094 raft_consensus.cc:3032] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:26.058460 4094 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:26.056656 4014 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.066186 2614 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:26.071820 4050 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.077754 3807 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.078441 3807 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.079242 3991 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:26.079528 4050 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.084633 2640 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:26.085088 1993 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.087046 1990 raft_consensus.cc:1226] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:26.092401 4094 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.092967 3988 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.093554 4094 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.096189 4156 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:26.058014 4154 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.096866 4156 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.097774 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.098515 2138 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:26.099022 4154 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.099469 4156 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.105720 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.106510 2066 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:26.080938 4020 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.107929 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.111588 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.083868 3993 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.079833 3991 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.115239 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.120546 2066 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:26.116782 1996 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:26.118631 2071 raft_consensus.cc:1261] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260924 00:22:26.113502 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.123886 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.124670 1996 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:26.126796 2066 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:26.130635 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:26.132251 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:26.135423 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 239852us
I20260924 00:22:26.145624 2066 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:26.146217 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.083567 4092 consensus_queue.cc:1035] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260924 00:22:26.110153 4154 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.110519 4156 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:26.109102 3940 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.118980 3529 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:26.152067 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 85997.9us
I20260924 00:22:26.168596 4112 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:26.169427 4112 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.160041 3988 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:26.125713 3993 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.161324 3807 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.187152 4167 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.190789 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.191843 2066 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:26.179253 3940 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.160740 4154 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.203796 4154 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.201078 2110 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:26.201561 4147 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.207892 2069 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:26.209049 1993 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:26.161661 4156 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.211956 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.212342 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.186228 4112 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.214257 2071 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:26.175501 3529 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.199672 4167 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.189559 3988 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.228855 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.230635 2136 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:26.230875 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.231530 2140 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:26.250304 2143 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:26.235055 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.251940 2071 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
I20260924 00:22:26.213102 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:22:26.236835 4090 consensus_meta.cc:287] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.713s user 0.001s sys 0.016s
I20260924 00:22:26.216156 4170 raft_consensus.cc:2781] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:26.224932 4167 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.221683 4107 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:26.226270 4145 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.224723 3993 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:26.250056 4122 consensus_meta.cc:287] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.675s user 0.014s sys 0.003s
I20260924 00:22:26.225900 3991 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.253449 3988 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.235958 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:26.209025 3940 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.223132 4147 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:26.262324 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 54861.9us
I20260924 00:22:26.264065 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.266247 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:26.261277 2646 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
W20260924 00:22:26.272244 2036 leader_election.cc:334] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.549s (SENT)
W20260924 00:22:26.279517 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 43637.2us
I20260924 00:22:26.273375 4122 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:26.342809 2065 raft_consensus.cc:2379] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:26.345039 3940 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.261004 4090 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:26.346201 3529 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.261682 4170 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:26.373358 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 107192us
I20260924 00:22:26.345402 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.263710 4107 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:26.383119 2613 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:26.383595 3993 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:26.374743 2036 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:26.393492 1995 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:22:26.400326 2614 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:26.394230 1987 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260924 00:22:26.394007 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 52589.7us
I20260924 00:22:26.376413 4166 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:26.401995 3940 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:26.404268 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:26.406693 4180 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:26.406893 4174 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:26.407864 4178 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.407581 4180 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.408357 4178 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.409319 3801 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.409873 3801 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.410391 4178 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.410574 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.404464 3940 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:26.407599 4174 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.411739 3940 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.411911 3801 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.413957 3807 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:26.406836 2641 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.415134 4100 raft_consensus.cc:3032] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:26.399609 3988 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.415563 4100 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:26.415829 3988 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.416361 4100 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.417568 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:26.418228 1996 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:26.405368 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 133148us
W20260924 00:22:26.419451 2036 leader_election.cc:334] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.588s (SENT)
I20260924 00:22:26.419813 2036 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:26.417519 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.423488 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.422781 1994 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:26.393513 4170 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:26.397939 3993 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:26.428514 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:26.429600 2110 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:26.399936 4107 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:26.407094 4145 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:26.429499 2647 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:26.435465 1996 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:26.435834 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:26.397469 4163 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260924 00:22:26.422240 4167 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:26.441979 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:26.426692 3807 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:26.423298 4174 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.430358 4180 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.444593 2649 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:26.431887 3993 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.436846 4145 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:26.458630 4145 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.454532 4091 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:26.462985 4091 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:26.463449 4091 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.439409 4156 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.448889 4167 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.456039 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:26.456564 1988 raft_consensus.cc:1226] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:26.446463 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:26.457871 4147 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:26.439284 4094 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.508359 3988 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.522747 2643 raft_consensus.cc:2379] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:26.523617 4094 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.526085 4094 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.521747 4147 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:26.500478 3807 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.527529 4147 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:26.538779 4179 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:26.532673 3807 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.542225 4179 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:26.551766 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 120544us
W20260924 00:22:26.551064 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 105423us
I20260924 00:22:26.553891 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:26.554761 2110 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:26.559816 2035 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:26.563469 1996 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:26.563827 1996 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:26.564864 2035 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:26.565515 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.564507 1996 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.568603 1996 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:26.574551 4159 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:26.575083 4159 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.575435 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.558889 4163 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:26.576510 4167 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.577085 4159 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.578207 4166 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:26.578625 4166 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:26.588438 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.589856 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:26.585567 4163 raft_consensus.cc:686] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:26.591683 4163 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:26.592437 4177 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.144s user 0.006s sys 0.000s
I20260924 00:22:26.586109 4178 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:26.593499 4178 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:26.593894 4178 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.587136 1996 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260924 00:22:26.599783 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 175648us
I20260924 00:22:26.560931 4182 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.603623 2613 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:26.567718 3988 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:26.607559 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.608573 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.608399 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.612486 2067 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:26.609489 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.610476 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.612197 2065 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:26.618409 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.581382 4174 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:26.619079 2068 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:26.624368 4174 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.625057 4145 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.626658 4174 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.596872 3991 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:26.627503 4145 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.580744 4186 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:26.621155 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.627333 3991 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.629943 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:26.572206 4020 consensus_queue.cc:1046] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
W20260924 00:22:26.622016 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 51898.3us
I20260924 00:22:26.631639 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:26.637611 2068 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.631816 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:26.614800 3988 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:26.642624 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 198907us
I20260924 00:22:26.645504 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:26.630093 4187 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:26.582516 4185 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260924 00:22:26.646082 1987 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.631134 4186 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:26.694742 4009 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:26.644120 3988 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.702270 3991 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.643666 2640 raft_consensus.cc:2354] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:26.641299 4115 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:26.710453 4148 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:26.711175 4174 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:26.714308 2068 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:26.639662 4145 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:26.631561 3993 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.710620 2641 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:26.612162 4182 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:26.722110 1960 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.288s user 0.013s sys 0.005s
I20260924 00:22:26.741822 1994 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:26.741925 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:26.743546 1960 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.310s user 0.015s sys 0.006s
I20260924 00:22:26.742479 2613 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:26.744074 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 1.31403e+06us
W20260924 00:22:26.740483 4157 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.862s user 0.008s sys 0.000s
I20260924 00:22:26.744557 4008 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.744917 1988 raft_consensus.cc:3032] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:26.745291 4097 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:26.736632 4174 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.745873 4097 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.747473 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:26.740895 4148 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.748797 4182 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.747517 4174 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.750344 4148 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.752080 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.752607 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.753744 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:26.753198 1996 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:26.718492 4186 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.755762 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:26.721688 4179 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.756951 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:26.760195 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:26.765714 1987 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:26.745327 1988 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:26.719533 4020 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260924 00:22:26.768061 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:26.720207 4091 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.751334 3807 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.775040 2110 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:26.768115 1988 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.708400 4193 consensus_queue.cc:1035] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260924 00:22:26.796157 1988 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.766276 4090 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:26.756598 4114 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:26.755156 4160 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:26.770126 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:26.780439 2036 leader_election.cc:334] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.682s (SENT)
I20260924 00:22:26.751158 3975 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
W20260924 00:22:26.773161 4105 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.952s user 0.013s sys 0.025s
I20260924 00:22:26.759007 4088 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:26.803972 4170 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.804265 3940 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:26.809449 4188 log.cc:924] Time spent T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
I20260924 00:22:26.773559 4179 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260924 00:22:26.804015 3993 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.815369 3988 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.802212 4175 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:26.767931 4097 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:26.825596 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 55567.1us
I20260924 00:22:26.847149 4175 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:26.847581 4175 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.847921 4088 consensus_queue.cc:1046] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:26.849159 4122 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:26.849527 4122 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:26.849661 4147 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.850313 4122 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.851698 4147 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.851938 4122 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:26.802505 4163 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:26.856200 2063 raft_consensus.cc:2379] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 6: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:26.857713 1988 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260924 00:22:26.835068 4105 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.015s user 0.016s sys 0.025s
I20260924 00:22:26.834836 3975 consensus_queue.cc:1225] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260924 00:22:26.768961 4161 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:26.830651 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 132) took 1824 ms. Trace:
I20260924 00:22:26.800220 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:26.796463 4178 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:26.817026 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:26.840987 4107 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:26.856840 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 88854.5us
I20260924 00:22:26.900763 2109 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:26.836681 4090 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:26.833397 4114 raft_consensus.cc:686] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:26.857759 1960 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:26.831107 2035 leader_election.cc:398] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:26.903103 2035 leader_election.cc:401] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260924 00:22:26.903647 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 83458.9us
I20260924 00:22:26.861581 1890 rpcz_store.cc:276] 0924 00:22:25.010500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:25.021586 (+ 11086us) service_pool.cc:227] Handling call
0924 00:22:26.830578 (+1808992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:25.209347 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:25.210032 (+ 685us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:25.210074 (+ 42us) write_op.cc:435] Acquired schema lock
0924 00:22:25.210112 (+ 38us) tablet.cc:660] Decoding operations
0924 00:22:25.210532 (+ 420us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:25.210629 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:25.210667 (+ 38us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:25.210925 (+ 258us) tablet.cc:699] Row locks acquired
0924 00:22:25.210957 (+ 32us) write_op.cc:260] PREPARE: finished
0924 00:22:25.211147 (+ 190us) write_op.cc:270] Start()
0924 00:22:25.211301 (+ 154us) write_op.cc:276] Timestamp: P: 1790209345211089 usec, L: 0
0924 00:22:25.211331 (+ 30us) op_driver.cc:331] REPLICATION: starting
0924 00:22:25.212135 (+ 804us) log.cc:842] Serialized 530 byte log entry
0924 00:22:25.781312 (+569177us) op_driver.cc:444] REPLICATION: finished
0924 00:22:26.196354 (+415042us) write_op.cc:301] APPLY: starting
0924 00:22:26.198854 (+ 2500us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:26.212701 (+ 13847us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:26.230785 (+ 18084us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:26.259548 (+ 28763us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:26.278337 (+ 18789us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:26.278407 (+ 70us) write_op.cc:312] APPLY: finished
0924 00:22:26.282261 (+ 3854us) log.cc:842] Serialized 23 byte log entry
0924 00:22:26.282479 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:26.282565 (+ 86us) write_op.cc:454] Released schema lock
0924 00:22:26.282615 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:26.282989 (+ 374us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:26.283026 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":392315,"apply.run_cpu_time_us":2898,"apply.run_wall_time_us":97530,"num_ops":1,"prepare.queue_time_us":100713,"prepare.run_cpu_time_us":3252,"prepare.run_wall_time_us":3253,"replication_time_us":555350,"thread_start_us":402765,"threads_started":2}]]}
I20260924 00:22:26.854697 3988 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:26.856388 4163 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:26.927659 4163 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:26.927438 4175 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:26.924336 4098 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:26.926719 3940 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:26.927219 2646 raft_consensus.cc:2379] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:26.933892 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:26.939051 2035 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:26.940452 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.941033 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.941833 2069 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:26.941067 2068 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:26.945515 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:26.946105 1987 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.946521 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.947086 2068 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:26.948360 1988 pending_rounds.cc:76] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 3
I20260924 00:22:26.948766 1988 pending_rounds.cc:97] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260924 00:22:26.950990 1988 raft_consensus.cc:2866] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:26.937446 4161 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:22:26.951989 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:26.965200 2143 raft_consensus.cc:595] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:26.923178 3993 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:26.956578 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.965891 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:26.959017 2066 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:26.950404 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:26.966626 1995 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:26.968447 2613 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:26.958781 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:26.966085 2069 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:26.969388 2067 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 4
I20260924 00:22:26.861820 4147 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.084s
I20260924 00:22:26.950497 3988 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:27.001041 2067 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:26.956557 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:27.009084 4174 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:27.015707 4108 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:27.001916 4149 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:26.964049 4091 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:26.952580 4193 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260924 00:22:27.002542 4147 consensus_queue.cc:1225] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260924 00:22:27.040323 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:26.963361 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:27.043340 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.055964 2136 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.044848 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.057184 2141 raft_consensus.cc:2354] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:27.057041 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.046123 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.009743 3988 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.057878 2144 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.059099 2139 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:27.065826 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.048272 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.034513 4182 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:26.960238 4166 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.066684 2142 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:27.071698 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:27.064479 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.057582 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.034500 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:27.038686 4108 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.047135 4175 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.049098 4174 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.079596 4174 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.080451 4178 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:27.013998 4114 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.064453 3988 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:27.086807 4170 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.074117 4182 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.098052 4182 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.098263 4166 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.088811 4186 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.103157 4186 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:26.959559 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 101908us
I20260924 00:22:27.137850 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:26.971625 1987 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:27.152624 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.154340 1989 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 1 FOLLOWER]: Advancing to term 3
I20260924 00:22:27.162155 1995 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:27.163889 1989 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:27.183547 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.184814 2071 raft_consensus.cc:2354] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:27.184939 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.185117 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.185927 2067 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:27.186170 2069 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:27.188815 2109 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:27.183753 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.191864 2065 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:27.198376 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.091776 4107 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.218603 2647 raft_consensus.cc:2379] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:27.212472 2614 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:27.203060 2067 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:27.202915 2069 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:27.207634 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.203961 2136 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:27.208302 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.215991 4097 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:27.209658 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.207391 2140 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:27.204780 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.253123 2139 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:27.221372 4090 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:27.255805 3988 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.255931 4090 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.252355 4097 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:27.257476 4097 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:27.257608 2065 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:27.248822 4160 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:27.217726 2644 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.104472 4163 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.253123 2137 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.251845 4108 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:27.266215 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 54179.7us
I20260924 00:22:27.292841 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.294451 2066 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:27.296561 4163 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.268133 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:27.268932 4170 raft_consensus.cc:2781] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:27.269865 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
W20260924 00:22:27.277424 4210 log.cc:924] Time spent T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.153s user 0.001s sys 0.001s
I20260924 00:22:27.306890 4170 raft_consensus.cc:686] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:27.280932 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.281679 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.309926 2067 raft_consensus.cc:2379] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.284132 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.315444 2063 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260924 00:22:27.315876 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.316339 2062 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:27.255841 4212 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:27.285197 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.286482 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.288196 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:27.320295 2070 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:27.302645 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.304257 4103 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:27.322717 2142 raft_consensus.cc:2379] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.320616 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.325696 2062 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:27.319038 2139 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.319778 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.329344 4090 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:27.333531 2067 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:27.329085 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.336053 2064 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:27.320341 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:27.308419 4170 consensus_queue.cc:237] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.343906 2641 raft_consensus.cc:2379] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 6: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.343873 4103 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.346438 2141 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 6: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.327965 4100 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:27.321798 4212 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.350247 4212 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.351181 4103 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.354683 4149 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:27.328452 2645 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
W20260924 00:22:27.357002 4122 consensus_meta.cc:287] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.505s user 0.001s sys 0.015s
I20260924 00:22:27.363003 3988 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.363711 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:27.366622 4166 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:27.377282 2068 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:27.378950 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:27.382413 2065 raft_consensus.cc:1226] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:27.381851 2064 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:27.368937 2071 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:27.390339 3988 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:27.385726 2067 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:27.398499 1994 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:27.404486 4098 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:27.409003 2143 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:27.409539 1892 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:27.416145 2035 leader_election.cc:398] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:27.416635 2035 leader_election.cc:401] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:27.417373 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:27.418869 2067 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:27.421365 4149 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:27.422061 4149 raft_consensus.cc:686] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:27.422344 3991 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4.
I20260924 00:22:27.422058 2035 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.397662 3988 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.424264 3988 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:27.424332 3807 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:27.424813 3807 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.425165 3807 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:27.414429 2640 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.427304 1994 pending_rounds.cc:76] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 1
I20260924 00:22:27.426574 3991 consensus_queue.cc:1046] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:27.427759 1994 pending_rounds.cc:97] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260924 00:22:27.428037 1994 raft_consensus.cc:2866] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:27.428666 4178 raft_consensus.cc:3032] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:27.429114 4178 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:27.429744 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33232 (request call id 217) took 3267 ms. Trace:
I20260924 00:22:27.430047 4178 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.431696 4178 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:27.433974 1892 catalog_manager.cc:5495] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 5, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:27.437633 1892 catalog_manager.cc:5495] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 3 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:27.439263 2035 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.430138 1891 rpcz_store.cc:276] 0924 00:22:24.162768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:24.162921 (+ 153us) service_pool.cc:227] Handling call
0924 00:22:27.429044 (+3266123us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:25.297263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:25.297934 (+ 671us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:25.297971 (+ 37us) write_op.cc:435] Acquired schema lock
0924 00:22:25.297990 (+ 19us) tablet.cc:660] Decoding operations
0924 00:22:25.435005 (+137015us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:25.440567 (+ 5562us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:25.462108 (+ 21541us) tablet.cc:683] Acquiring locks for 15 operations
0924 00:22:25.559784 (+ 97676us) tablet.cc:699] Row locks acquired
0924 00:22:25.559806 (+ 22us) write_op.cc:260] PREPARE: finished
0924 00:22:25.559951 (+ 145us) write_op.cc:270] Start()
0924 00:22:25.560076 (+ 125us) write_op.cc:276] Timestamp: P: 1790209345559911 usec, L: 0
0924 00:22:25.560108 (+ 32us) op_driver.cc:331] REPLICATION: starting
0924 00:22:25.621749 (+ 61641us) log.cc:842] Serialized 5884 byte log entry
0924 00:22:26.596516 (+974767us) op_driver.cc:444] REPLICATION: finished
0924 00:22:26.621880 (+ 25364us) write_op.cc:301] APPLY: starting
0924 00:22:26.631003 (+ 9123us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:26.719151 (+ 88148us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:26.751029 (+ 31878us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:26.806093 (+ 55064us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:26.806350 (+ 257us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0924 00:22:26.806396 (+ 46us) write_op.cc:312] APPLY: finished
0924 00:22:26.806748 (+ 352us) log.cc:842] Serialized 107 byte log entry
0924 00:22:26.806908 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:26.807066 (+ 158us) write_op.cc:454] Released schema lock
0924 00:22:26.807120 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:26.807361 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:26.807392 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34233,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":4226,"apply.run_cpu_time_us":4723,"apply.run_wall_time_us":190399,"num_ops":15,"prepare.queue_time_us":24361,"prepare.run_cpu_time_us":6710,"prepare.run_wall_time_us":414978,"replication_time_us":1036307}]]}
I20260924 00:22:27.453125 2071 raft_consensus.cc:1261] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:27.445246 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.426908 2068 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:27.448553 1990 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:27.448951 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.450801 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.450827 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.451947 4122 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:27.452442 4212 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:27.452765 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.455726 2066 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:27.440619 4149 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.453125 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.459048 1988 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:27.459925 4122 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.461076 1989 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:27.462212 4122 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.463294 1995 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.462183 2110 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:27.465934 3991 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:27.468408 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.468428 3991 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.469730 3991 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:27.462512 4212 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.474102 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.474535 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:27.475044 2067 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:27.475706 4212 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.475585 2064 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:27.476334 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:27.477514 2035 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:27.477581 4119 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:27.477874 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.478514 4174 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:27.478464 4119 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:27.479167 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.479254 4174 raft_consensus.cc:686] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:27.480432 4174 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.483815 4163 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:27.484305 4163 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.484686 4163 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:27.489385 1892 catalog_manager.cc:5495] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 3 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:27.493301 4119 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.497532 4173 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:27.498193 4173 raft_consensus.cc:686] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:27.499220 4173 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.455883 4097 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.502157 2145 raft_consensus.cc:2379] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.505650 2109 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:27.506734 4193 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:27.507109 4193 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.507444 4193 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:27.506765 2613 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:27.510766 4149 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:27.517431 4189 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20260924 00:22:27.523484 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:27.503386 4100 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:27.465751 3807 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.525537 2110 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:27.528545 4189 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:27.528898 1994 raft_consensus.cc:1226] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:27.529229 4189 raft_consensus.cc:686] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:27.530395 4189 consensus_queue.cc:237] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.531580 4098 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:27.531987 4098 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:27.532991 4098 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:27.524603 4097 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:27.525769 4100 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.542024 1992 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:27.533740 3991 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.543009 4100 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:27.549820 3807 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:27.551172 4214 log.cc:924] Time spent T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260924 00:22:27.552809 3988 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.555606 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.541273 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.559470 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.563556 4163 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.566510 2110 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:27.568919 4097 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:27.571568 4105 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:27.572091 4105 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.572482 4185 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:27.574234 4105 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.574290 2110 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:27.626403 2064 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:22:27.591614 1988 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:27.628876 4185 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:27.596541 1989 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:22:27.619839 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.585943 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.630059 4149 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:27.616140 4178 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:27.631206 1994 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:27.631564 4178 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.629258 4185 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.631646 4170 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:27.632429 4185 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:27.586544 1992 raft_consensus.cc:1203] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Deduplicated request from leader. Original: 6.4->[7.5-7.5] Dedup: 7.5->[]
I20260924 00:22:27.634613 1992 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260924 00:22:27.586894 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.637688 3988 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:27.638417 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:27.588238 1990 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:27.575515 4097 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:27.605643 3991 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:27.642279 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 68075.1us
W20260924 00:22:27.591410 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 58665.3us
I20260924 00:22:27.592159 4103 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:27.656250 2110 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:27.596681 4193 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.668155 4103 raft_consensus.cc:686] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:27.669384 4103 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.670567 4185 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.672613 4173 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:27.672713 4185 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.669957 4193 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.641541 4097 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.674612 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.675956 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.676633 4105 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:27.677495 4105 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.679642 2143 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:27.680751 4105 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.681253 3807 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:27.681905 3807 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.682363 4178 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.683938 3807 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:27.650869 4100 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.645669 4163 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.692955 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.695343 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.696393 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:27.696653 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.698012 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.701570 1994 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:27.703570 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.673087 4173 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.702490 2141 raft_consensus.cc:2354] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4.
I20260924 00:22:27.698940 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.612006 4098 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.716934 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:27.717835 2066 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:27.711462 3807 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:27.713070 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.689518 3991 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:27.724611 3991 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.726624 3991 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.715729 2071 raft_consensus.cc:1226] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:27.730952 2035 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.718947 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.732766 4178 raft_consensus.cc:2741] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260924 00:22:27.722219 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.734045 3807 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:27.733197 1994 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:27.736554 3807 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:27.731987 2035 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.737058 4098 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:27.737788 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.741351 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:27.741993 2068 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:27.742261 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.742422 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.743610 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.743691 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.744388 1995 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.744674 1987 raft_consensus.cc:2379] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:27.744789 4178 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:27.745450 4178 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:27.745364 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.745983 4178 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:27.746907 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.747162 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.745843 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.748486 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.748600 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.749540 1988 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:27.749971 1994 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.750690 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.752480 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:27.752549 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.719662 4185 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20260924 00:22:27.754917 2140 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:27.756857 3807 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:27.757771 3991 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:27.759120 3991 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.761881 3991 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:27.762097 3807 raft_consensus.cc:686] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:27.763087 3807 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.766870 2035 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.769516 3991 raft_consensus.cc:2741] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260924 00:22:27.774855 2136 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:27.775501 2145 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:27.777462 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.780793 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.706676 4173 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:27.794831 2640 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:27.795989 2137 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:27.796103 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.797370 1989 raft_consensus.cc:2379] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:27.799573 2142 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:27.807973 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:27.808461 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.808928 2644 raft_consensus.cc:2379] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:27.809772 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.811882 2139 raft_consensus.cc:2379] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.811312 2035 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:27.814253 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.815973 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.817324 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.785645 4178 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:27.819509 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.818260 3991 raft_consensus.cc:2741] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260924 00:22:27.824815 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.825914 2139 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.827457 2140 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:27.828774 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.829980 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.830032 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.832902 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.830914 1996 raft_consensus.cc:2362] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5. Re-sending same reply.
I20260924 00:22:27.838194 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:27.839977 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.846271 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.845902 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.848866 1996 raft_consensus.cc:2379] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:27.850739 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:27.856383 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.851315 4100 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:27.869014 2036 leader_election.cc:334] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.727s (SENT)
I20260924 00:22:27.869760 2036 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:27.871425 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.872771 1989 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:27.879824 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.881654 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.878284 4105 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:22:27.894806 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.917075 1989 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
W20260924 00:22:27.925412 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 134579us
I20260924 00:22:27.926121 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.926997 1996 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:27.929479 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.929914 2109 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:27.913954 4178 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:27.932883 2036 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.639s (SENT) [suppressed 21 similar messages]
I20260924 00:22:27.942957 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:27.945891 2140 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:27.947850 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:27.936225 4097 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:27.950814 2065 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:27.953537 2143 raft_consensus.cc:1226] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:27.953727 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.958846 2068 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:27.953756 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.969774 1996 raft_consensus.cc:2379] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:27.971901 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:27.986562 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:27.987308 2068 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:28.002420 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:28.003439 2065 raft_consensus.cc:2379] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
W20260924 00:22:28.004140 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 85467.7us
I20260924 00:22:28.006736 2069 raft_consensus.cc:1226] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:28.007257 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.024178 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.027186 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.027943 1994 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:28.006745 4170 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260924 00:22:28.083832 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.107044 1994 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:28.108631 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:28.110101 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 114021us
I20260924 00:22:28.119771 2068 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:28.123370 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:28.130698 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.131542 1988 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:28.127244 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:28.133744 2143 raft_consensus.cc:595] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:28.134114 2143 raft_consensus.cc:603] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260924 00:22:28.136974 4228 log.cc:924] Time spent T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.192s user 0.001s sys 0.000s
I20260924 00:22:28.138278 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:28.108569 4163 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260924 00:22:28.144369 2068 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:28.152575 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:28.154170 2066 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:22:28.154392 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.157369 1993 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:28.163187 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.163769 1990 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:28.153631 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:28.169480 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.174652 2063 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:28.173012 1995 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:28.176606 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.179046 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:28.180925 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.182039 1987 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:22:28.191938 1990 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:28.193531 4178 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:28.183245 3988 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:28.198297 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.197868 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:28.201421 1995 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:28.202342 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:28.203943 2068 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:28.206863 2110 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:28.159300 4240 consensus_queue.cc:1046] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260924 00:22:28.214466 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.215831 1988 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 00:22:28.223652 4173 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.227600 2110 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:28.228395 2035 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:28.230579 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 119115us
I20260924 00:22:28.204581 3807 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:28.235704 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.232120 4149 raft_consensus.cc:3032] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:28.241869 2110 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:28.240097 4178 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:28.241786 4149 raft_consensus.cc:729] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:28.245448 4149 consensus_queue.cc:260] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:28.247915 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33238 (request call id 230) took 3203 ms. Trace:
I20260924 00:22:28.234740 4193 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:28.234890 4158 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:28.257989 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.232792 4170 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260924 00:22:28.264762 1890 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:28.269449 2644 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:22:28.269876 4173 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:28.242637 4103 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:28.257501 4158 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:28.271381 4158 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:28.270833 4103 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:28.273926 4103 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:28.251515 4098 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:28.278079 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.279388 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.280607 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.281493 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.281795 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.282404 1994 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 8
I20260924 00:22:28.284474 2143 raft_consensus.cc:1226] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:28.285405 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.285656 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.286219 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.287045 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.288645 2109 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:28.289762 1987 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:28.294124 1992 raft_consensus.cc:3032] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:28.294449 1992 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:28.295198 1992 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:28.296190 1992 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:28.297499 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.293951 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:28.298233 1990 raft_consensus.cc:2379] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:28.298024 1994 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:28.298686 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:28.308547 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:28.308825 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:28.309525 2064 raft_consensus.cc:2354] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8.
W20260924 00:22:28.317582 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 66930.6us
I20260924 00:22:28.318768 2110 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:28.344003 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:28.251873 1892 rpcz_store.cc:276] 0924 00:22:25.044719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:25.044914 (+ 195us) service_pool.cc:227] Handling call
0924 00:22:28.247856 (+3202942us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:27.660746 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:27.661489 (+ 743us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:27.661537 (+ 48us) write_op.cc:435] Acquired schema lock
0924 00:22:27.661556 (+ 19us) tablet.cc:660] Decoding operations
0924 00:22:27.662192 (+ 636us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:27.662272 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:27.662310 (+ 38us) tablet.cc:683] Acquiring locks for 4 operations
0924 00:22:27.662651 (+ 341us) tablet.cc:699] Row locks acquired
0924 00:22:27.662674 (+ 23us) write_op.cc:260] PREPARE: finished
0924 00:22:27.662793 (+ 119us) write_op.cc:270] Start()
0924 00:22:27.662896 (+ 103us) write_op.cc:276] Timestamp: P: 1790209347662743 usec, L: 0
0924 00:22:27.662923 (+ 27us) op_driver.cc:331] REPLICATION: starting
0924 00:22:27.663599 (+ 676us) log.cc:842] Serialized 1682 byte log entry
0924 00:22:27.857424 (+193825us) op_driver.cc:444] REPLICATION: finished
0924 00:22:28.117646 (+260222us) write_op.cc:301] APPLY: starting
0924 00:22:28.128018 (+ 10372us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:28.165352 (+ 37334us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:28.190428 (+ 25076us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:28.191131 (+ 703us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:28.191426 (+ 295us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0924 00:22:28.191482 (+ 56us) write_op.cc:312] APPLY: finished
0924 00:22:28.191811 (+ 329us) log.cc:842] Serialized 41 byte log entry
0924 00:22:28.192011 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:28.192102 (+ 91us) write_op.cc:454] Released schema lock
0924 00:22:28.192157 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:28.192442 (+ 285us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:28.192471 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":17558,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":230258,"apply.run_cpu_time_us":3076,"apply.run_wall_time_us":83458,"num_ops":4,"prepare.queue_time_us":146743,"prepare.run_cpu_time_us":3272,"prepare.run_wall_time_us":3271,"replication_time_us":191218,"thread_start_us":269637,"threads_started":2}]]}
I20260924 00:22:28.350612 1991 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 6
I20260924 00:22:28.306167 4185 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:28.299862 3991 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260924 00:22:28.357216 4178 raft_consensus.cc:686] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:28.396420 4119 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:28.395673 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:28.394752 4185 raft_consensus.cc:686] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:28.419679 4119 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:28.501585 4119 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.505936 4178 consensus_queue.cc:237] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.512565 1992 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:22:28.578773 1991 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260924 00:22:28.598024 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 708537d593c14817accda67f9fd081b4, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:28.598493 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 708537d593c14817accda67f9fd081b4, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:28.600896 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 253809us
I20260924 00:22:28.696033 4149 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
W20260924 00:22:28.726630 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 406583us
I20260924 00:22:28.743254 3940 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:28.743866 3940 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:28.746325 3940 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:28.719184 4163 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
W20260924 00:22:28.763857 2036 leader_election.cc:334] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.843s (SENT)
I20260924 00:22:28.714313 3807 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:28.771085 2110 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:28.792740 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:28.733919 4170 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:28.804466 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:28.676817 4185 consensus_queue.cc:237] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.818400 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:28.827206 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 149159us
I20260924 00:22:28.814703 4260 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260924 00:22:28.866784 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 127158us
I20260924 00:22:28.856285 4163 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:28.865815 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:28.881413 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:28.881740 4265 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:28.882444 2143 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 8: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:28.882437 4265 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:28.883972 4265 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.862068 4166 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:28.885648 4166 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:28.887447 4166 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:28.890245 2136 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260924 00:22:28.892161 4158 consensus_meta.cc:287] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.620s user 0.010s sys 0.001s
I20260924 00:22:28.892634 4158 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.894696 4158 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:28.868115 4103 consensus_meta.cc:287] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.594s user 0.011s sys 0.000s
I20260924 00:22:28.894797 2641 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:28.814857 4170 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:28.905858 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 95046us
I20260924 00:22:28.819893 4258 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260924 00:22:28.815129 3807 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.913647 2649 raft_consensus.cc:1226] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:28.929397 4163 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:28.933439 4246 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:28.933717 4163 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:28.934523 4170 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:28.934271 4246 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:28.935740 4163 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:28.898869 4103 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:28.943303 4103 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:28.864488 4097 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
W20260924 00:22:28.963672 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 102572us
I20260924 00:22:28.988042 1989 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:28.973327 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:28.990206 1987 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:29.020422 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 193686us
I20260924 00:22:28.991310 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:29.033106 4166 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:29.034415 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:29.034885 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:29.035465 1988 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:29.036720 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:29.037688 1991 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:28.958045 4260 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.056728 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:29.071169 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 107902us
I20260924 00:22:29.075922 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:29.086689 4256 consensus_queue.cc:1046] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260924 00:22:29.019955 4178 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:29.106787 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.131975 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.118851 4178 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:29.147300 4259 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.403s user 0.000s sys 0.014s
I20260924 00:22:29.149936 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1576) took 1079 ms. Trace:
I20260924 00:22:29.150461 1996 rpcz_store.cc:276] 0924 00:22:28.070345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:28.070462 (+ 117us) service_pool.cc:227] Handling call
0924 00:22:28.083213 (+ 12751us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:28.261458 (+178245us) raft_consensus.cc:1520] Early marking committed up to index 6
0924 00:22:28.322473 (+ 61015us) log.cc:842] Serialized 14 byte log entry
0924 00:22:28.603148 (+280675us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:28.603586 (+ 438us) log.cc:842] Serialized 26 byte log entry
0924 00:22:28.603705 (+ 119us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:22:28.603750 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:28.603817 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:29.146599 (+542782us) spinlock_profiling.cc:243] Waited 67.1 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:29.149529 (+ 2930us) raft_consensus.cc:1671] finished
0924 00:22:29.149676 (+ 147us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:29.149863 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105780,"prepare.queue_time_us":83447,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":56,"raft.queue_time_us":106485,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":58,"spinlock_wait_cycles":154393984,"thread_start_us":266959,"threads_started":2,"wal-append.queue_time_us":391329}
W20260924 00:22:29.154672 4261 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.412s user 0.004s sys 0.000s
I20260924 00:22:29.156849 4246 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:29.157091 4263 log.cc:924] Time spent T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.420s user 0.004s sys 0.006s
I20260924 00:22:29.067862 4260 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.160688 3807 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.162047 4178 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:29.168958 4252 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.419s user 0.001s sys 0.005s
I20260924 00:22:29.174885 4260 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.176348 2613 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:29.180752 2613 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:29.183382 4114 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:29.181092 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.190433 1988 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:29.182126 4246 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:29.172756 1987 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:29.222087 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 133) took 1730 ms. Trace:
I20260924 00:22:29.217999 4178 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.222263 3940 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.226070 4246 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.223798 1890 rpcz_store.cc:276] 0924 00:22:27.489699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:27.489955 (+ 256us) service_pool.cc:227] Handling call
0924 00:22:29.219558 (+1729603us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:28.744955 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:28.745489 (+ 534us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:28.745526 (+ 37us) write_op.cc:435] Acquired schema lock
0924 00:22:28.745539 (+ 13us) tablet.cc:660] Decoding operations
0924 00:22:28.745846 (+ 307us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:28.745900 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:28.745927 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:28.746098 (+ 171us) tablet.cc:699] Row locks acquired
0924 00:22:28.746111 (+ 13us) write_op.cc:260] PREPARE: finished
0924 00:22:28.746257 (+ 146us) write_op.cc:270] Start()
0924 00:22:28.746363 (+ 106us) write_op.cc:276] Timestamp: P: 1790209348746200 usec, L: 0
0924 00:22:28.746379 (+ 16us) op_driver.cc:331] REPLICATION: starting
0924 00:22:28.746973 (+ 594us) log.cc:842] Serialized 530 byte log entry
0924 00:22:28.932531 (+185558us) op_driver.cc:444] REPLICATION: finished
0924 00:22:29.137387 (+204856us) write_op.cc:301] APPLY: starting
0924 00:22:29.137509 (+ 122us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:29.137639 (+ 130us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:29.137793 (+ 154us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:29.138085 (+ 292us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:29.138294 (+ 209us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:29.138367 (+ 73us) write_op.cc:312] APPLY: finished
0924 00:22:29.138595 (+ 228us) log.cc:842] Serialized 23 byte log entry
0924 00:22:29.138795 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:29.138899 (+ 104us) write_op.cc:454] Released schema lock
0924 00:22:29.138950 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:29.139308 (+ 358us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:29.139356 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26277,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":199730,"apply.run_cpu_time_us":2275,"apply.run_wall_time_us":2275,"num_ops":1,"prepare.queue_time_us":476362,"prepare.run_cpu_time_us":2268,"prepare.run_wall_time_us":2269,"replication_time_us":186058,"thread_start_us":304990,"threads_started":2}]]}
I20260924 00:22:29.268929 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.270088 2139 raft_consensus.cc:2354] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:29.270591 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.272336 2145 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:29.278286 1992 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:29.278666 2140 raft_consensus.cc:1226] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:29.271181 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.298216 2143 raft_consensus.cc:2354] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8.
W20260924 00:22:29.273633 4149 consensus_meta.cc:287] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Time spent flushing consensus metadata: real 0.577s user 0.006s sys 0.006s
I20260924 00:22:29.297726 4256 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:29.251822 3940 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.273679 4246 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.237579 4178 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:29.314414 4253 log.cc:924] Time spent T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.482s user 0.006s sys 0.012s
I20260924 00:22:29.334937 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:29.343190 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 223965us
I20260924 00:22:29.318691 4260 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:29.314356 4256 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.332629 4158 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.352341 4149 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:22:29.378400 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 43552.7us
I20260924 00:22:29.363547 4286 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:29.387316 2110 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:29.358302 4260 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.390892 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 300) took 1122 ms. Trace:
I20260924 00:22:29.363159 4158 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.401808 2110 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:29.380398 4193 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:29.392717 1995 rpcz_store.cc:276] 0924 00:22:28.268800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:28.277414 (+ 8614us) service_pool.cc:227] Handling call
0924 00:22:28.277820 (+ 406us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:28.278143 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:28.278170 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:28.278561 (+ 391us) log.cc:842] Serialized 26 byte log entry
0924 00:22:28.592329 (+313768us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:28.592379 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:28.592449 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:29.184864 (+592415us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:29.385653 (+200789us) raft_consensus.cc:1671] finished
0924 00:22:29.385686 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:29.390823 (+ 5137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1035,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"spinlock_wait_cycles":152064896,"thread_start_us":119577,"threads_started":1,"wal-append.queue_time_us":466570}
I20260924 00:22:29.367826 4255 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260924 00:22:29.401019 4100 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260924 00:22:29.410386 4158 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:29.411015 4279 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260924 00:22:29.413800 4248 log.cc:924] Time spent T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.677s user 0.003s sys 0.010s
I20260924 00:22:29.414592 4246 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:29.415807 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 1474) took 1286 ms. Trace:
I20260924 00:22:29.416043 4257 raft_consensus.cc:3032] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:29.416546 4257 raft_consensus.cc:729] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:29.417749 4285 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.417407 4257 consensus_queue.cc:260] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.418550 4257 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:29.418256 4285 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:29.419139 4272 log.cc:924] Time spent T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.345s user 0.000s sys 0.009s
W20260924 00:22:29.418010 4276 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.343s user 0.000s sys 0.009s
I20260924 00:22:29.421666 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:29.370282 4149 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.428723 1995 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:22:29.430095 2614 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:29.426752 4284 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.431756 4284 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.434578 4285 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.435010 4282 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.435729 4282 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.416203 2071 rpcz_store.cc:276] 0924 00:22:28.129632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:28.129814 (+ 182us) service_pool.cc:227] Handling call
0924 00:22:28.149849 (+ 20035us) raft_consensus.cc:1457] Updating replica for 2 ops
0924 00:22:28.184819 (+ 34970us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:28.184837 (+ 18us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0924 00:22:28.215334 (+ 30497us) log.cc:842] Serialized 52 byte log entry
0924 00:22:28.287012 (+ 71678us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:28.306921 (+ 19909us) log.cc:842] Serialized 14 byte log entry
0924 00:22:28.315873 (+ 8952us) log.cc:842] Serialized 14 byte log entry
0924 00:22:28.343693 (+ 27820us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:28.343817 (+ 124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:28.612382 (+268565us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b18003791e0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d1b94638 00007fb6d1b97b11 00007fb6d1b97ac9 00007fb6d1b97a59 00007fb6d1b97a21 00007fb6d1b96d1d 00007fb6dadf8204 00007fb6dadf8039 00007fb6d1b76dc6
0924 00:22:29.403342 (+790960us) spinlock_profiling.cc:243] Waited 74.9 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:29.415582 (+ 12240us) raft_consensus.cc:1671] finished
0924 00:22:29.415605 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:29.415732 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":107001,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":3265,"raft.queue_time_us":48423,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":20241,"spinlock_wait_cycles":204525184,"thread_start_us":41936,"threads_started":1,"wal-append.queue_time_us":503962}
I20260924 00:22:29.454751 4282 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:29.455801 4260 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.457274 4284 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.459615 4178 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.460731 4292 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.461333 4158 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:29.426669 4163 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:29.438747 4170 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.478756 1990 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:29.481472 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:29.482959 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:29.483301 2142 raft_consensus.cc:1261] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:29.483718 4244 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:29.483705 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.484647 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.485534 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.485447 1893 catalog_manager.cc:5495] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 1 to 3, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 3 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:29.543604 1991 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:29.551245 2145 raft_consensus.cc:595] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:29.518329 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:29.535110 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 56582.2us
I20260924 00:22:29.487007 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:29.535197 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 47384.2us
I20260924 00:22:29.519138 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.554422 2136 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:29.446005 4114 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.556825 1990 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:29.523448 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.558902 1994 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:29.404016 4286 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.461283 4292 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.477581 3940 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:29.606032 4266 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.421s user 0.000s sys 0.002s
I20260924 00:22:29.566820 1990 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.607874 1990 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:29.607124 1893 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 7, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:29.552809 4244 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:29.613654 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 59569.7us
I20260924 00:22:29.557875 2138 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:29.523924 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.521611 4293 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:29.561884 4163 raft_consensus.cc:686] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:29.504244 4284 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:29.621592 4259 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260924 00:22:29.627925 2036 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:29.637555 4284 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.563465 4170 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.618010 4256 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.644157 1990 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:29.643563 1893 catalog_manager.cc:5495] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:29.553077 4178 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.074s
I20260924 00:22:29.645494 2035 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:29.622825 4244 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:29.558614 4114 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:29.632408 4254 log.cc:924] Time spent T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.648s user 0.002s sys 0.010s
W20260924 00:22:29.641983 4290 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20260924 00:22:29.655903 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:29.656436 2140 raft_consensus.cc:595] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:29.656747 2140 raft_consensus.cc:603] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:29.652592 2110 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:29.653829 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 166719us
I20260924 00:22:29.631434 4105 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:29.657781 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.635721 4147 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:29.662743 1996 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:29.567937 3807 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.085s
I20260924 00:22:29.649435 2640 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
W20260924 00:22:29.676229 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 48425us
I20260924 00:22:29.671913 1960 leader_election.cc:398] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4.
I20260924 00:22:29.676985 1960 leader_election.cc:401] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260924 00:22:29.647711 3940 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:29.685763 3940 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.686414 4299 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:29.686967 4299 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.646143 4293 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.688365 4158 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:29.688278 4163 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.693840 4299 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.695077 4293 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.694973 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.683007 1893 catalog_manager.cc:5495] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:29.684849 4147 raft_consensus.cc:686] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:29.697299 4114 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.698046 4147 consensus_queue.cc:237] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:29.698768 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.698954 4244 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.699438 2140 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:22:29.685187 4286 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.689644 2036 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:29.714028 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:29.682381 4268 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:29.692709 1960 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:29.693290 4285 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4.
I20260924 00:22:29.709101 2613 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:29.660245 4256 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:29.714849 1992 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:29.714205 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:29.661161 4178 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260924 00:22:29.720036 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 74618.3us
I20260924 00:22:29.724507 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:29.682278 4291 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:29.726111 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:29.699550 4170 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.659782 4282 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260924 00:22:29.707098 4149 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.661031 4287 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.325s
I20260924 00:22:29.671617 4105 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:29.682310 4289 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.123s
I20260924 00:22:29.807512 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:29.927251 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 234638us
I20260924 00:22:29.826014 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 308) took 1590 ms. Trace:
I20260924 00:22:29.732052 1992 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:29.988087 2143 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:22:29.943751 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:29.989711 2614 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:29.869516 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 179956us
I20260924 00:22:29.795320 4257 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:29.995220 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:29.764664 4256 raft_consensus.cc:686] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:29.861649 4292 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:30.022662 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:29.986792 2063 rpcz_store.cc:276] 0924 00:22:28.235530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:28.235687 (+ 157us) service_pool.cc:227] Handling call
0924 00:22:28.241860 (+ 6173us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:28.257113 (+ 15253us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:28.257174 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:28.263562 (+ 6388us) log.cc:842] Serialized 26 byte log entry
0924 00:22:28.375527 (+111965us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:28.375574 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:28.375655 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:29.301707 (+926052us) spinlock_profiling.cc:243] Waited 66.8 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:29.804320 (+502613us) raft_consensus.cc:1671] finished
0924 00:22:29.804342 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:29.825938 (+ 21596us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15247,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":7305,"spinlock_wait_cycles":153730304,"thread_start_us":89545,"threads_started":1,"wal-append.queue_time_us":451244}
I20260924 00:22:29.989727 2138 raft_consensus.cc:595] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:30.298050 2138 raft_consensus.cc:603] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:29.752666 1893 catalog_manager.cc:5495] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 2 to 4, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:30.098877 4303 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:29.805027 3940 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.028913 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:30.300323 4303 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.303686 4303 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:29.807292 4271 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:29.943996 4178 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:30.304857 4271 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:30.305123 4178 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:30.074131 4299 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:30.306514 4299 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.305435 4271 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:30.110983 2036 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.303399 1893 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
W20260924 00:22:30.311466 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 239479us
I20260924 00:22:30.312495 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:29.864504 2640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 332) took 1059 ms. Trace:
I20260924 00:22:30.313786 2640 rpcz_store.cc:276] 0924 00:22:28.805373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:28.806073 (+ 700us) service_pool.cc:227] Handling call
0924 00:22:29.629336 (+823263us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:29.705957 (+ 76621us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:29.864271 (+158314us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b540050f850. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abba09 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:29.864414 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1849262336}
I20260924 00:22:30.313627 1893 catalog_manager.cc:5495] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
W20260924 00:22:29.804252 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 50460.9us
W20260924 00:22:30.317791 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 322640us
W20260924 00:22:29.931787 4297 log.cc:924] Time spent T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
W20260924 00:22:30.316426 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 293808us
I20260924 00:22:30.318688 2110 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:29.719821 4286 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:30.333107 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:29.862241 4294 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260924 00:22:30.337760 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 199758us
I20260924 00:22:29.808365 4269 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:30.334045 2613 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.331213 4178 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:29.921350 4244 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:30.325440 1893 catalog_manager.cc:5495] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
W20260924 00:22:30.144822 4302 log.cc:924] Time spent T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.283s user 0.003s sys 0.000s
W20260924 00:22:29.995332 4304 log.cc:924] Time spent T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260924 00:22:30.318475 4257 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.319600 4287 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:30.320024 4289 raft_consensus.cc:2741] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260924 00:22:29.826282 4293 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:30.289314 4291 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:30.020763 4147 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:30.347153 4269 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:30.341579 4241 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:30.309053 4299 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:29.860307 4284 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.320103 3940 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:30.370189 3940 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:30.370685 3940 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:30.337204 4303 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:30.355351 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:30.020535 4149 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:30.360174 4291 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:30.351497 4286 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:30.361786 4256 consensus_queue.cc:237] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:30.363909 4105 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:30.350252 4260 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:29.753899 4268 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.352753 4250 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:30.398298 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:30.402995 2036 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.374044 4173 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260924 00:22:30.463920 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 65728.1us
I20260924 00:22:30.374974 4098 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:30.373970 4269 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:30.380153 4299 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:30.401383 4284 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:30.463984 4286 raft_consensus.cc:686] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:30.463109 4149 raft_consensus.cc:686] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:30.468919 4292 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:30.466862 4173 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:30.467439 4273 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:30.506800 1960 leader_election.cc:398] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:30.470110 4105 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:30.375267 4147 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.398239 4293 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.517917 1960 leader_election.cc:401] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:30.464530 4260 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.526728 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:30.488734 4158 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:30.523170 2641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45582 (request call id 282) took 1076 ms. Trace:
I20260924 00:22:30.507689 4313 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:30.500094 4314 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:30.535180 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:30.540499 2143 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:30.514839 4173 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:30.478014 4098 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.489782 4299 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.519616 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:30.562505 1987 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260924 00:22:30.524883 4292 raft_consensus.cc:686] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:30.571848 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:22:30.581795 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 205201us
I20260924 00:22:30.583365 1988 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:30.536728 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:22:30.469496 4250 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.534461 4283 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:30.529080 4105 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.589027 4178 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.535881 4158 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.614360 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.529232 4273 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:30.615289 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 85720.5us
I20260924 00:22:30.616551 1960 leader_election.cc:398] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8.
I20260924 00:22:30.616858 1960 leader_election.cc:401] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:30.596141 4257 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:30.566404 4289 consensus_queue.cc:1046] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260924 00:22:30.530895 4306 consensus_queue.cc:1046] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
I20260924 00:22:30.595001 4284 raft_consensus.cc:686] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:30.636601 2036 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.642136 4268 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.642740 4268 raft_consensus.cc:3032] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:30.643077 4268 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:30.643563 4282 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:30.643870 4268 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.645227 4282 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260924 00:22:30.650898 4105 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:30.651367 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260924 00:22:30.645577 4307 log.cc:924] Time spent T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260924 00:22:30.652139 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:30.652963 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1626) took 1062 ms. Trace:
I20260924 00:22:30.653363 1989 rpcz_store.cc:276] 0924 00:22:29.590840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:29.591029 (+ 189us) service_pool.cc:227] Handling call
0924 00:22:29.625828 (+ 34799us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:29.633946 (+ 8118us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:29.633991 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:29.643084 (+ 9093us) log.cc:842] Serialized 26 byte log entry
0924 00:22:30.133356 (+490272us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:30.133412 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:30.133506 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:30.652759 (+519253us) raft_consensus.cc:1671] finished
0924 00:22:30.652784 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:30.652910 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":95,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":1870,"thread_start_us":472457,"threads_started":1,"wal-append.queue_time_us":755334}
W20260924 00:22:30.645886 4310 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20260924 00:22:30.655068 4306 raft_consensus.cc:3032] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:30.655437 4306 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:30.656188 4306 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.621531 4178 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:30.532385 4291 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.662451 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:30.663051 4291 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:30.663108 1994 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
W20260924 00:22:30.635623 4312 log.cc:924] Time spent T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260924 00:22:30.628182 4283 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8.
I20260924 00:22:30.595574 4286 consensus_queue.cc:237] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.668648 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:30.646196 4309 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260924 00:22:30.532598 2641 rpcz_store.cc:276] 0924 00:22:29.443524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:29.445057 (+ 1533us) service_pool.cc:227] Handling call
0924 00:22:29.458743 (+ 13686us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:29.506547 (+ 47804us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:29.524696 (+ 18149us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:29.606936 (+ 82240us) log.cc:842] Serialized 26 byte log entry
0924 00:22:29.649247 (+ 42311us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:29.660615 (+ 11368us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:29.672072 (+ 11457us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:30.144010 (+471938us) spinlock_profiling.cc:243] Waited 9.53 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:22:30.489099 (+345089us) raft_consensus.cc:1671] finished
0924 00:22:30.493224 (+ 4125us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:30.517342 (+ 24118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4240,"raft.queue_time_us":19762,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":98,"spinlock_wait_cycles":21917056,"thread_start_us":28743,"threads_started":1,"wal-append.queue_time_us":157547}
W20260924 00:22:30.669821 4317 log.cc:924] Time spent T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260924 00:22:30.669655 4284 consensus_queue.cc:237] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:30.670588 4319 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260924 00:22:30.665199 4260 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.553673 4313 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.674795 4303 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:30.673293 4105 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:30.634688 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:30.703408 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 1623) took 1161 ms. Trace:
W20260924 00:22:30.679593 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 152240us
I20260924 00:22:30.645107 4268 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:30.647512 4158 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:30.682009 2110 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:30.678272 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:30.646752 4293 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.672214 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:30.687120 4326 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:30.617378 4149 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.668887 4299 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.727914 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:30.689793 4250 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:30.691453 4273 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:30.693536 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:30.701742 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:30.746282 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:30.746490 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:30.687230 4306 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:30.708976 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:30.703873 1993 rpcz_store.cc:276] 0924 00:22:29.541826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:29.541990 (+ 164us) service_pool.cc:227] Handling call
0924 00:22:29.567584 (+ 25594us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:29.621521 (+ 53937us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:29.621549 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:29.633339 (+ 11790us) log.cc:842] Serialized 26 byte log entry
0924 00:22:29.990068 (+356729us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:29.990105 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:29.990172 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:30.703151 (+712979us) raft_consensus.cc:1671] finished
0924 00:22:30.703177 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:30.703330 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21999,"raft.queue_time_us":58687,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4616,"thread_start_us":338885,"threads_started":1,"wal-append.queue_time_us":840149}
I20260924 00:22:30.674587 4098 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:30.715569 4262 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:30.757083 4262 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:30.757529 4262 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:30.716786 3940 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.716365 4105 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:30.719792 4271 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:30.766160 4271 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.723977 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:30.724824 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:30.729063 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:30.729676 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:30.646438 4292 consensus_queue.cc:237] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.722114 4313 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:30.741909 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:30.747902 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:30.749557 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 66099.5us
I20260924 00:22:30.777115 1960 leader_election.cc:398] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:30.777576 1960 leader_election.cc:401] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260924 00:22:30.752063 4293 raft_consensus.cc:2741] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
W20260924 00:22:30.759312 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 50442.4us
I20260924 00:22:30.783881 2035 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:30.715636 2036 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.705732 4303 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.728724 4147 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:30.768208 4299 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:30.771884 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:30.768555 1987 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:30.775612 2142 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:30.789981 2641 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 8: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:30.790256 3940 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:30.790513 4326 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6.
I20260924 00:22:30.795851 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:30.796695 2137 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:30.799494 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 83932.2us
I20260924 00:22:30.800462 2036 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.787025 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:30.806066 2110 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:30.807857 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:30.798868 4105 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:30.847623 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:30.848414 2142 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
W20260924 00:22:30.850898 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 51682.3us
W20260924 00:22:30.868892 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 62938.4us
I20260924 00:22:30.810249 4257 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:30.810433 4299 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:30.829291 4313 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:30.846112 4105 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:22:30.964330 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 89379.3us
I20260924 00:22:30.968412 2142 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:30.868561 4173 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.980093 2613 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:30.981540 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 179774us
I20260924 00:22:30.873941 4315 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:30.983803 4286 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260924 00:22:30.988233 4315 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:30.991699 4315 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:30.975809 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 99614.9us
I20260924 00:22:31.003670 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:31.004410 2143 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:30.982386 2036 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:30.979744 4105 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:31.003850 4280 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:31.001616 4170 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:31.048854 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.002030 4286 consensus_queue.cc:1046] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260924 00:22:31.057423 4178 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:31.059106 1988 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:31.061507 2137 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:31.020646 4173 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:30.981778 4269 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.025547 4287 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:31.026489 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:31.033303 4299 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 00:22:31.033968 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 51704us
I20260924 00:22:31.035030 4303 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.041693 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.066166 2613 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:31.078733 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 52320.5us
I20260924 00:22:31.047669 4280 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:31.053062 4170 raft_consensus.cc:686] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [term 8 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:31.088805 4097 raft_consensus.cc:979] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:31.088092 4178 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:31.112309 4178 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:31.112742 4178 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.115012 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.117290 2036 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:31.117436 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 67492.8us
I20260924 00:22:31.118475 2036 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:31.054145 4262 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.120561 4170 consensus_queue.cc:237] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:31.122510 2036 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:31.124087 2036 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:31.091528 4286 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:31.088632 4280 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:31.127969 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 86410.9us
I20260924 00:22:31.130009 1960 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.132043 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.132784 2136 raft_consensus.cc:2379] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:31.115957 3940 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:31.163036 1988 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:31.125295 4303 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:31.127643 4299 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:31.127498 4313 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:31.127306 4322 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:31.184262 2036 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:31.159950 4287 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:31.172080 4284 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260924 00:22:31.191115 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 72054.8us
I20260924 00:22:31.179133 3940 raft_consensus.cc:686] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
W20260924 00:22:31.206794 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 74879.6us
I20260924 00:22:31.209232 4103 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:31.211510 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.191604 4255 raft_consensus.cc:1067] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:31.199374 4289 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:31.214916 1960 leader_election.cc:398] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8.
I20260924 00:22:31.203275 4299 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:31.198988 4262 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:31.216272 4284 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:31.232143 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:31.178982 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 67894.3us
I20260924 00:22:31.238052 3940 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:31.204133 4322 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:31.235535 4337 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260924 00:22:31.234653 4269 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:31.259137 2140 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:31.221969 4103 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.262951 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:31.272822 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.276670 4322 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:31.276576 4178 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:31.278759 4178 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:31.268054 4284 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:31.280252 4105 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.281186 4103 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:31.288959 1988 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260924 00:22:31.289620 4346 log.cc:924] Time spent T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20260924 00:22:31.290335 4331 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20260924 00:22:31.291946 4287 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:31.292011 4313 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:31.271531 4299 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:31.293255 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.325053 2063 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:31.313109 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.314435 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.315966 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:31.275467 2036 connection.cc:232] RPC call timeout handler was delayed by -1.6375442579999344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1574s for a response
W20260924 00:22:31.329312 2036 connection.cc:232] RPC call timeout handler was delayed by -1.5382708749998528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
W20260924 00:22:31.329631 2036 connection.cc:232] RPC call timeout handler was delayed by -1.5218504990000383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
W20260924 00:22:31.329912 2036 connection.cc:232] RPC call timeout handler was delayed by -1.3946301519997633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
W20260924 00:22:31.330349 2036 connection.cc:232] RPC call timeout handler was delayed by -1.340544257999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
W20260924 00:22:31.323783 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 113559us
W20260924 00:22:31.330677 2036 connection.cc:232] RPC call timeout handler was delayed by -1.0601038729998891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1636s for a response
I20260924 00:22:31.333613 2067 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:31.336386 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.337227 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.337388 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.338191 2071 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:31.336304 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.337989 2067 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:31.339933 2035 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:31.341246 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.341912 2071 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.342429 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:31.350847 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.346496 1988 raft_consensus.cc:1261] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260924 00:22:31.352170 1988 pending_rounds.cc:76] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 3
I20260924 00:22:31.352385 1988 pending_rounds.cc:97] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260924 00:22:31.352607 1988 raft_consensus.cc:2866] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:31.343425 2139 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 8: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:31.349670 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.354753 2063 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:31.356648 2062 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.362277 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.353607 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:31.354187 4303 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:31.325223 4244 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260924 00:22:31.371995 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.373210 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.375982 1987 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:31.377199 1990 raft_consensus.cc:1226] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260924 00:22:31.327598 4287 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.336014 4299 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.383314 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.362797 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:31.386831 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.327296 4313 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:31.383937 2063 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:31.387601 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.389927 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.391100 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.394385 4105 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.374675 4303 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:31.394634 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.395027 4303 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:31.395689 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.396673 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.398211 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.398824 1996 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:31.399538 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.399878 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:31.400511 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.400734 4284 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:31.401203 4284 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:31.401628 4284 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.402411 4306 raft_consensus.cc:3032] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:31.402325 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.402849 4306 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.403532 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.403649 4306 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.404551 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.404817 4306 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:31.405560 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:31.406611 4105 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:22:31.410768 4284 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.413399 4280 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.415699 2143 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:31.421531 4299 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.423707 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.424314 2063 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:31.416757 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:31.436492 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet aa5e2f99029440bd9d42ec2a01200d4a, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:31.436905 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet aa5e2f99029440bd9d42ec2a01200d4a, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.442337 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.412954 4284 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.455680 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.445909 1996 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:31.457809 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.458281 2070 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260924 00:22:31.460664 1990 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:31.460934 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.461623 1992 raft_consensus.cc:2379] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:31.456868 1996 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:31.463749 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.464151 1996 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:31.464316 1990 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:31.477636 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.483433 2070 raft_consensus.cc:1261] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260924 00:22:31.485447 2136 raft_consensus.cc:1226] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:31.491786 4313 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:31.457417 4280 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:31.502995 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:31.503737 2066 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:31.506145 2136 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:31.506219 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.507128 2139 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:31.510049 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.511451 1996 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.513897 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.529245 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.530018 2137 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:31.511811 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.530865 2063 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:31.531077 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.533385 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.534833 1996 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.536271 2136 raft_consensus.cc:1226] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:31.536360 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:31.539788 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:31.520666 4361 log.cc:924] Time spent T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260924 00:22:31.544176 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.512650 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.534337 1993 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:31.541051 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.536617 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:31.549675 2140 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:31.555187 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.556530 2068 raft_consensus.cc:1226] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:31.558773 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:31.559353 2144 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:31.559587 2144 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:31.560667 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.560514 4289 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.566082 2070 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.565770 4268 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:31.566373 2067 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:31.566917 2144 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.571058 2144 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:31.526120 4287 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.586839 2066 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:31.561347 4178 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:31.554309 4105 consensus_queue.cc:1046] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260924 00:22:31.582405 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.588465 2070 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.588820 4105 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:31.589329 4268 raft_consensus.cc:3032] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:31.589514 4105 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:31.589780 4268 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.590484 4105 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.591820 4105 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:31.578904 4286 consensus_queue.cc:1046] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:31.584815 2144 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:31.586921 2064 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.590627 4268 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.582957 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.593786 4284 raft_consensus.cc:3032] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:31.608322 4284 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.609205 4284 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.610088 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.610357 4284 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:31.610783 2062 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:31.611053 2066 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:31.612267 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.584568 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.613425 2071 raft_consensus.cc:3032] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:31.613845 2071 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:31.613924 2063 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:22:31.583755 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:31.591910 2068 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:31.614535 2071 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:31.615483 2071 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:31.612628 2064 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:31.604137 2070 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:31.622678 2140 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:31.623385 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:31.635036 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.639218 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.642144 2035 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:31.643857 2065 raft_consensus.cc:2379] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.644039 2071 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
W20260924 00:22:31.615231 4366 log.cc:924] Time spent T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260924 00:22:31.644551 2071 pending_rounds.cc:76] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 2
I20260924 00:22:31.644608 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.647082 2071 pending_rounds.cc:97] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260924 00:22:31.647428 2071 raft_consensus.cc:2866] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:31.647900 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:31.648749 4292 raft_consensus.cc:3032] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:31.649098 4292 raft_consensus.cc:729] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.649251 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:31.651760 2138 raft_consensus.cc:1226] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:31.649777 4292 consensus_queue.cc:260] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.654443 4306 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:31.654814 4292 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:31.655309 4306 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:31.655735 4306 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:31.656116 4306 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:31.654358 4343 consensus_queue.cc:1046] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:31.657747 4343 raft_consensus.cc:3032] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:31.658085 4343 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:31.658896 4343 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.660137 4343 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.660257 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.660859 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.661584 2064 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.661453 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.661754 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.662122 2068 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:31.662520 2065 raft_consensus.cc:2379] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.662398 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.663925 4292 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:31.675760 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.678035 2068 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:31.682408 2071 raft_consensus.cc:2379] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.682487 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.683646 2068 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:31.685838 2110 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:31.689002 4193 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:31.653121 4256 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:31.689489 4193 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:31.689927 4256 raft_consensus.cc:686] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:31.633287 4286 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:31.690590 4286 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:31.690958 4256 consensus_queue.cc:237] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.693526 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.696599 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.699286 4289 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260924 00:22:31.700773 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 51642us
I20260924 00:22:31.701637 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.689940 4193 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.688166 4306 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.710269 4306 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.711944 2065 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:31.670665 4268 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:31.713099 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.714581 4306 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:31.716969 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:31.721379 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.724150 2068 raft_consensus.cc:2362] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7. Re-sending same reply.
I20260924 00:22:31.621429 4303 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.690984 4286 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:31.726238 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.728796 2071 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:31.732671 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.722853 4306 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:31.733942 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.736864 2070 raft_consensus.cc:2354] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5.
I20260924 00:22:31.737715 2071 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:31.728401 4289 consensus_queue.cc:1046] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260924 00:22:31.756490 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:31.766482 2138 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 5
I20260924 00:22:31.743671 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:31.743367 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.769591 2145 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:31.771327 3940 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:31.750563 4193 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.754225 4303 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:31.749630 4284 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:31.773854 4193 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:31.774456 4292 raft_consensus.cc:3032] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:31.774866 4292 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.775631 4292 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.776496 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.776819 4292 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:31.777119 1989 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:31.755577 4289 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:31.786193 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.786891 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.786821 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.787102 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.791675 2065 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.788131 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.792554 2064 raft_consensus.cc:2379] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:31.788712 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.789108 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.787743 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.793625 2071 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:31.791428 2070 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:22:31.799513 2066 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:31.800592 1892 catalog_manager.cc:5495] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 5 to 8, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 8 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:31.803449 2138 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:31.803332 1892 catalog_manager.cc:5495] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 5 to 6, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:31.806334 2035 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:31.820533 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 86664us
I20260924 00:22:31.797370 4306 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.819036 1989 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:31.822844 2138 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:31.822346 1960 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:31.805132 1892 catalog_manager.cc:5495] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 3 to 6, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 6 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:31.832662 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.834923 2110 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:31.805559 4289 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.813544 4303 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:31.841886 2069 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:31.846108 2066 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:31.840718 4373 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:31.842612 4359 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:31.851202 1892 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:31.858181 2138 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:31.863922 4359 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:31.864909 4359 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.851358 4303 raft_consensus.cc:686] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:31.872145 1989 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:31.863044 4373 raft_consensus.cc:686] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:31.876372 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.876113 4373 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.876998 2068 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:31.878958 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:31.879448 2068 raft_consensus.cc:1261] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:31.868027 4170 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260924 00:22:31.885803 4303 consensus_queue.cc:237] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.873234 4306 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.893245 4277 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:31.901795 2109 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.901180 1892 catalog_manager.cc:5495] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 1 to 3, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 3 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:31.903335 1892 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:31.904425 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.898272 4289 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.906476 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.917330 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:31.906804 4277 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:31.927807 1996 raft_consensus.cc:1226] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
W20260924 00:22:31.936378 2036 leader_election.cc:334] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.574s (SENT)
I20260924 00:22:31.936861 2036 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:31.921087 4343 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:31.940202 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.944142 2613 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.947480 4166 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:31.947667 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:31.948391 1996 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:31.947697 1990 raft_consensus.cc:2396] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260924 00:22:31.937597 4105 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:31.955137 4286 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:31.945062 4271 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:31.920524 3940 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260924 00:22:31.958170 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:31.962563 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33246 (request call id 217) took 6716 ms. Trace:
I20260924 00:22:31.967046 1994 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:31.962971 1893 rpcz_store.cc:276] 0924 00:22:25.245838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:25.257330 (+ 11492us) service_pool.cc:227] Handling call
0924 00:22:31.962466 (+6705136us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:30.668717 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:30.669358 (+ 641us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:30.669408 (+ 50us) write_op.cc:435] Acquired schema lock
0924 00:22:30.669419 (+ 11us) tablet.cc:660] Decoding operations
0924 00:22:30.670464 (+ 1045us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:30.670515 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:30.670545 (+ 30us) tablet.cc:683] Acquiring locks for 8 operations
0924 00:22:30.674050 (+ 3505us) tablet.cc:699] Row locks acquired
0924 00:22:30.674066 (+ 16us) write_op.cc:260] PREPARE: finished
0924 00:22:30.674203 (+ 137us) write_op.cc:270] Start()
0924 00:22:30.674303 (+ 100us) write_op.cc:276] Timestamp: P: 1790209350674161 usec, L: 0
0924 00:22:30.674322 (+ 19us) op_driver.cc:331] REPLICATION: starting
0924 00:22:30.678235 (+ 3913us) log.cc:842] Serialized 3210 byte log entry
0924 00:22:31.010755 (+332520us) op_driver.cc:444] REPLICATION: finished
0924 00:22:31.201537 (+190782us) write_op.cc:301] APPLY: starting
0924 00:22:31.204723 (+ 3186us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:31.235632 (+ 30909us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:31.248453 (+ 12821us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:31.250563 (+ 2110us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:31.250760 (+ 197us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0924 00:22:31.250804 (+ 44us) write_op.cc:312] APPLY: finished
0924 00:22:31.251048 (+ 244us) log.cc:842] Serialized 65 byte log entry
0924 00:22:31.251243 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:31.251360 (+ 117us) write_op.cc:454] Released schema lock
0924 00:22:31.251401 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:31.251646 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:31.251673 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":991,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":172885,"apply.run_cpu_time_us":3566,"apply.run_wall_time_us":60619,"num_ops":8,"prepare.queue_time_us":262880,"prepare.run_cpu_time_us":16688,"prepare.run_wall_time_us":89064,"raft.queue_time_us":256599,"raft.run_cpu_time_us":7911,"raft.run_wall_time_us":116641,"replication_time_us":336315,"thread_start_us":465313,"threads_started":4,"wal-append.queue_time_us":71996}]]}
I20260924 00:22:31.982995 4280 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:31.984318 4103 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:31.985397 4258 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:31.984951 2109 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:31.984930 4103 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:31.989917 2109 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:31.984272 4097 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20260924 00:22:31.998067 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 61726.5us
W20260924 00:22:31.998799 2036 leader_election.cc:334] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.589s (SENT)
W20260924 00:22:31.999603 2036 leader_election.cc:334] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.599s (SENT)
I20260924 00:22:32.000036 2036 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.005615 2109 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:32.011727 2036 leader_election.cc:334] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.636s (SENT)
I20260924 00:22:32.012681 2109 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:32.015489 2036 leader_election.cc:334] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.904s (SENT)
I20260924 00:22:31.998036 4274 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:32.017439 1996 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:31.989374 4103 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.005048 4258 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.019491 2109 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:32.003800 4100 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:32.021559 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 74858.6us
I20260924 00:22:32.011108 4105 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:32.029343 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:32.030184 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.014420 4321 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:22:32.032311 4098 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:32.001005 4343 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260924 00:22:32.028681 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:32.022833 4274 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:32.040036 2109 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:32.026923 4286 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:32.014806 4277 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260924 00:22:32.036338 3940 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:32.046963 2109 leader_election.cc:398] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5.
I20260924 00:22:32.047470 2109 leader_election.cc:401] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:32.030640 4359 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:22:32.038443 4354 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.040203 4274 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:32.054653 4193 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5.
I20260924 00:22:32.035710 4100 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.040083 4103 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:32.041178 4098 raft_consensus.cc:686] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:32.039369 4271 consensus_queue.cc:1035] T b76aee4623d7484e8ff6c1f1a13f5fea P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.004s
W20260924 00:22:32.093361 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 65896.9us
I20260924 00:22:32.045406 3940 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:32.049688 4097 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:32.090392 4286 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:32.094857 2036 leader_election.cc:334] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 2.096s (SENT)
I20260924 00:22:32.096796 2036 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:32.099135 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 59101.8us
I20260924 00:22:32.096421 3940 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:32.110347 2140 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:32.104226 4303 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:32.126511 4250 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:32.127282 4315 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:32.130753 2140 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:32.099033 4286 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.135588 1996 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
W20260924 00:22:32.153380 4383 log.cc:924] Time spent T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260924 00:22:32.118664 4313 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260924 00:22:32.158340 4286 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:32.130331 4098 consensus_queue.cc:237] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.160003 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:32.163681 4280 consensus_queue.cc:1046] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:32.164659 4280 consensus_queue.cc:1225] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260924 00:22:32.165925 4258 raft_consensus.cc:3032] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:32.166316 4258 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:32.166229 2036 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:32.168097 2036 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.168829 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:32.170019 4250 consensus_queue.cc:1035] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:32.171466 4250 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:32.171846 4250 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:32.172210 4250 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:32.167193 4258 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.173776 4258 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:32.173851 4100 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:32.174966 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:32.176724 4286 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.177250 4257 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:32.169620 4303 consensus_queue.cc:1046] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260924 00:22:32.192239 2145 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:32.182503 4303 raft_consensus.cc:3032] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:32.194494 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:32.203658 2036 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.209751 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.209575 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:32.211289 1990 raft_consensus.cc:2396] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260924 00:22:32.211546 4286 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:32.212252 4286 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:32.214704 4315 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:32.213065 4286 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.226699 4274 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.230014 4315 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:32.231472 2036 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.233084 3940 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:32.254001 4315 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.255689 3940 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:32.237504 4274 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:32.194491 4303 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:32.248879 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:32.266477 4250 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.266425 1989 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:32.268754 4250 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:32.274978 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:32.275856 2066 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:32.278821 1989 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:32.274387 4303 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:32.290010 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 189267us
I20260924 00:22:32.297221 4303 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:32.299010 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:32.253036 2036 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:32.302076 2110 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:32.302377 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 49411.7us
I20260924 00:22:32.303930 2036 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.305627 4103 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:32.306277 4103 raft_consensus.cc:686] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:32.306429 4313 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.308640 2036 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:32.309345 1989 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:32.313169 4256 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:32.314236 4256 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:32.333472 4103 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.307812 4286 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260924 00:22:32.328526 4303 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:32.353901 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 45324.5us
I20260924 00:22:32.354990 2036 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.368924 2640 raft_consensus.cc:1226] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:32.327726 4359 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260924 00:22:32.379361 4256 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:32.393345 2066 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:32.394507 4274 consensus_queue.cc:1046] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:32.395406 4274 raft_consensus.cc:3032] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:32.395721 4274 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:32.403710 4286 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:32.361670 4303 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.396490 4274 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:32.412395 4303 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:32.415505 4274 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:32.414865 1890 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:22:32.408874 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 53967.8us
I20260924 00:22:32.420670 1890 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 7 to 8, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:32.434084 2036 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.435601 1890 catalog_manager.cc:5495] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 2 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:32.443333 1890 catalog_manager.cc:5495] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 1 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:32.445061 4105 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:32.448635 2145 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:32.448796 4105 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.450995 4105 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:32.451968 4286 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:32.445636 4402 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:32.447054 2036 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:32.456118 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 146199us
I20260924 00:22:32.469732 2036 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.470166 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:32.471738 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.475395 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:32.476394 2138 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:32.454711 4402 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:32.481925 2036 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.484548 2035 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:32.485916 2036 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.476326 4303 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:32.491070 2140 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:32.481356 4256 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:32.487082 4250 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.498679 4303 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:32.511125 4400 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:32.537089 2069 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:32.552369 2069 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:32.530172 4303 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:32.512552 4286 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
W20260924 00:22:32.564757 4404 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260924 00:22:32.565174 4292 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:32.566628 4303 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
W20260924 00:22:32.605188 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 119383us
I20260924 00:22:32.607933 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:32.608593 2062 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:32.613628 2066 raft_consensus.cc:1203] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
I20260924 00:22:32.614225 2066 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260924 00:22:32.626322 2138 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:32.647329 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:32.649293 2063 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:32.655139 4391 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:32.667492 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:32.689294 2062 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:32.689810 2063 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:32.641660 4292 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260924 00:22:32.717028 1990 raft_consensus.cc:1261] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260924 00:22:32.717443 1990 pending_rounds.cc:76] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 2
I20260924 00:22:32.717732 1990 pending_rounds.cc:97] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260924 00:22:32.717996 1990 raft_consensus.cc:2866] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:32.720180 2063 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:32.731930 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:32.733211 4286 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:32.763907 4286 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.748652 4303 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.765857 4286 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:32.766486 2067 raft_consensus.cc:2379] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:32.767138 4303 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:32.772116 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 133917us
I20260924 00:22:32.774255 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:32.775290 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:32.775772 1993 raft_consensus.cc:2362] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8. Re-sending same reply.
I20260924 00:22:32.777109 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:32.778003 1995 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260924 00:22:32.778697 1995 pending_rounds.cc:76] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 3
I20260924 00:22:32.778577 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:32.778996 1995 pending_rounds.cc:97] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260924 00:22:32.779330 1995 raft_consensus.cc:2866] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:32.779770 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:32.778025 1992 raft_consensus.cc:1261] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:32.780512 1991 raft_consensus.cc:2396] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260924 00:22:32.782217 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:32.782689 1988 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:32.771664 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:32.774907 2068 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:32.792786 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:32.793321 2145 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:32.797791 2036 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.799147 4299 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:32.799412 2036 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.799839 4299 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:32.801261 2036 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:32.801677 4313 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.802415 4315 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:32.803241 4315 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:32.804494 4303 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 00:22:32.804890 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:32.810206 4299 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.606858 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:32.846429 2070 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:32.851095 1988 raft_consensus.cc:2396] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260924 00:22:32.828435 4166 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:32.845898 1892 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 4 to 6, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 6 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260924 00:22:32.860795 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 54432.4us
I20260924 00:22:32.762859 4363 consensus_queue.cc:1035] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
W20260924 00:22:32.854884 4417 log.cc:924] Time spent T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260924 00:22:32.873502 4414 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:32.924886 2070 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260924 00:22:32.942044 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 268552us
W20260924 00:22:32.948071 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 143244us
W20260924 00:22:32.965637 4416 log.cc:924] Time spent T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260924 00:22:32.978076 4292 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:22:32.986356 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 43621.5us
W20260924 00:22:32.995822 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 46511.6us
I20260924 00:22:32.916059 4414 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:32.952500 4434 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:33.033852 2140 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260924 00:22:33.007997 4413 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
I20260924 00:22:32.978878 4292 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:33.038878 4292 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:33.040026 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:33.040813 2064 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:33.046772 4354 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:33.051753 2109 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:33.053123 4258 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:33.058532 2064 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:33.059576 4434 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:33.060045 4434 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:33.067625 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:33.068475 2140 raft_consensus.cc:595] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:33.074069 2140 raft_consensus.cc:1261] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:33.075757 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:33.084086 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33238 (request call id 231) took 3690 ms. Trace:
I20260924 00:22:33.094049 2067 raft_consensus.cc:1261] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:33.094331 4403 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:33.080861 4414 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:33.084524 1890 rpcz_store.cc:276] 0924 00:22:29.403807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:29.407221 (+ 3414us) service_pool.cc:227] Handling call
0924 00:22:33.084022 (+3676801us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:32.490734 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:32.521577 (+ 30843us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:32.521645 (+ 68us) write_op.cc:435] Acquired schema lock
0924 00:22:32.521666 (+ 21us) tablet.cc:660] Decoding operations
0924 00:22:32.522432 (+ 766us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:32.522500 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:32.522540 (+ 40us) tablet.cc:683] Acquiring locks for 4 operations
0924 00:22:32.523026 (+ 486us) tablet.cc:699] Row locks acquired
0924 00:22:32.523047 (+ 21us) write_op.cc:260] PREPARE: finished
0924 00:22:32.523228 (+ 181us) write_op.cc:270] Start()
0924 00:22:32.523354 (+ 126us) write_op.cc:276] Timestamp: P: 1790209352523177 usec, L: 0
0924 00:22:32.523386 (+ 32us) op_driver.cc:331] REPLICATION: starting
0924 00:22:32.524114 (+ 728us) log.cc:842] Serialized 1682 byte log entry
0924 00:22:32.536814 (+ 12700us) op_driver.cc:444] REPLICATION: finished
0924 00:22:32.560461 (+ 23647us) write_op.cc:301] APPLY: starting
0924 00:22:32.560556 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:32.564552 (+ 3996us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:32.564657 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:32.565307 (+ 650us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:32.565485 (+ 178us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0924 00:22:32.565533 (+ 48us) write_op.cc:312] APPLY: finished
0924 00:22:32.565794 (+ 261us) log.cc:842] Serialized 41 byte log entry
0924 00:22:32.570887 (+ 5093us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:32.572516 (+ 1629us) write_op.cc:454] Released schema lock
0924 00:22:32.573230 (+ 714us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:32.574637 (+ 1407us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:32.574680 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7533,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":11396,"apply.run_cpu_time_us":2751,"apply.run_wall_time_us":14508,"num_ops":4,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":3701,"prepare.run_wall_time_us":44517,"replication_time_us":8085}]]}
I20260924 00:22:33.125648 4433 consensus_queue.cc:1035] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260924 00:22:33.127277 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 84336.4us
I20260924 00:22:33.128158 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:33.131735 4430 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:33.132262 4430 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:33.132696 4430 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:33.134720 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:33.135468 2064 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:33.139902 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:33.177402 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:33.178252 2143 raft_consensus.cc:2379] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 6: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:33.173738 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:33.181185 2138 raft_consensus.cc:595] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:33.161672 4292 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
W20260924 00:22:33.265941 4413 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260924 00:22:33.169121 4423 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.146s
W20260924 00:22:33.282646 4407 log.cc:924] Time spent T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260924 00:22:33.468523 1990 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260924 00:22:33.386447 4415 log.cc:924] Time spent T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260924 00:22:33.514129 4258 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:33.525000 2140 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:33.616236 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:33.624259 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 150474us
W20260924 00:22:33.634799 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 159513us
W20260924 00:22:33.635183 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 159419us
W20260924 00:22:33.645689 4439 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20260924 00:22:33.630836 2144 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:22:33.648073 4441 log.cc:924] Time spent T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260924 00:22:33.691365 2035 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:33.698840 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 57477.3us
W20260924 00:22:33.717268 4443 log.cc:924] Time spent T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260924 00:22:33.699391 4250 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election won for term 8
W20260924 00:22:33.728751 4434 consensus_meta.cc:287] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.668s user 0.010s sys 0.000s
I20260924 00:22:33.792024 4434 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:33.766898 4430 consensus_meta.cc:287] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.593s user 0.007s sys 0.005s
I20260924 00:22:33.755416 4436 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:33.798553 4436 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:33.798902 4436 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:33.708823 4098 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
W20260924 00:22:33.756856 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 578710us
I20260924 00:22:33.793541 4250 raft_consensus.cc:686] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:33.804380 4250 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:33.758692 2140 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260924 00:22:33.796886 4430 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:33.812536 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 122755us
W20260924 00:22:33.702961 4437 log.cc:924] Time spent T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260924 00:22:33.822960 4434 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:33.825688 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 338) took 1119 ms. Trace:
I20260924 00:22:33.826133 2066 rpcz_store.cc:276] 0924 00:22:32.706769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:32.756293 (+ 49524us) service_pool.cc:227] Handling call
0924 00:22:32.756632 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:22:32.760544 (+ 3912us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:32.760585 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:22:32.769374 (+ 8789us) log.cc:842] Serialized 26 byte log entry
0924 00:22:33.319926 (+550552us) raft_consensus.cc:1631] Marking committed up to 7
0924 00:22:33.516964 (+197038us) log.cc:842] Serialized 14 byte log entry
0924 00:22:33.517192 (+ 228us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:33.517270 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:22:33.825491 (+308221us) raft_consensus.cc:1671] finished
0924 00:22:33.825510 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:33.825627 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":161176,"prepare.queue_time_us":184914,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":3688,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":2568,"thread_start_us":482715,"threads_started":1,"wal-append.queue_time_us":745821}
I20260924 00:22:33.932595 1989 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:33.932688 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ff940e15e1849efb194c908520c212f"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:33.933851 2140 raft_consensus.cc:595] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:33.934423 2140 raft_consensus.cc:603] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260924 00:22:33.937043 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 70975.6us
I20260924 00:22:33.951378 4250 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:33.952296 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 135) took 1546 ms. Trace:
I20260924 00:22:33.858134 4434 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:33.953534 4400 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:33.965586 2062 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:33.967363 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:33.892093 4432 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20260924 00:22:33.909225 4421 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:33.886976 4464 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:33.858394 4430 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:33.975689 1988 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260924 00:22:33.884013 4438 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:33.909497 4402 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:33.978741 1893 catalog_manager.cc:5495] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 5, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:33.991798 1893 catalog_manager.cc:5495] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:33.983593 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:33.961753 1892 rpcz_store.cc:276] 0924 00:22:32.405909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:32.406823 (+ 914us) service_pool.cc:227] Handling call
0924 00:22:33.952236 (+1545413us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:33.456572 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:33.457168 (+ 596us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:33.457209 (+ 41us) write_op.cc:435] Acquired schema lock
0924 00:22:33.457222 (+ 13us) tablet.cc:660] Decoding operations
0924 00:22:33.457572 (+ 350us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:33.457632 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:33.457671 (+ 39us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:33.468655 (+ 10984us) tablet.cc:699] Row locks acquired
0924 00:22:33.468673 (+ 18us) write_op.cc:260] PREPARE: finished
0924 00:22:33.485456 (+ 16783us) write_op.cc:270] Start()
0924 00:22:33.499396 (+ 13940us) write_op.cc:276] Timestamp: P: 1790209353481881 usec, L: 0
0924 00:22:33.499433 (+ 37us) op_driver.cc:331] REPLICATION: starting
0924 00:22:33.561805 (+ 62372us) log.cc:842] Serialized 530 byte log entry
0924 00:22:33.786665 (+224860us) op_driver.cc:444] REPLICATION: finished
0924 00:22:33.930333 (+143668us) write_op.cc:301] APPLY: starting
0924 00:22:33.930410 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:33.930507 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:33.930624 (+ 117us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:33.930834 (+ 210us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:33.930982 (+ 148us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:33.931022 (+ 40us) write_op.cc:312] APPLY: finished
0924 00:22:33.946617 (+ 15595us) log.cc:842] Serialized 23 byte log entry
0924 00:22:33.946844 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:33.946955 (+ 111us) write_op.cc:454] Released schema lock
0924 00:22:33.947013 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:33.947323 (+ 310us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:33.947349 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":487341,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":142631,"apply.run_cpu_time_us":2189,"apply.run_wall_time_us":18029,"num_ops":1,"prepare.queue_time_us":369211,"prepare.run_cpu_time_us":3305,"prepare.run_wall_time_us":105555,"replication_time_us":283610,"thread_start_us":298912,"threads_started":2}]]}
I20260924 00:22:33.986034 4438 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:33.989965 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 43771.9us
I20260924 00:22:33.994417 1893 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 6 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:34.001309 2066 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:33.999984 1893 catalog_manager.cc:5495] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:33.968868 4400 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:34.004029 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.005049 1893 catalog_manager.cc:5495] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:34.007561 1893 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 6, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:33.972065 4434 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.018167 1893 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
W20260924 00:22:34.019388 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 44807.4us
I20260924 00:22:34.019475 1893 catalog_manager.cc:5495] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } }
I20260924 00:22:34.013610 4430 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:34.029024 1988 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:34.033332 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.006265 4403 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260924 00:22:34.034139 2137 raft_consensus.cc:2379] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 6: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:33.989645 4421 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.035358 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.035971 2145 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:34.040910 2145 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:33.992327 4464 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.013018 4436 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.046938 4436 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.052866 4456 raft_consensus.cc:979] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:34.053079 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.054246 4411 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:34.054773 4411 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.056908 4411 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.033655 4430 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.059942 4430 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.064940 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:33.998071 4402 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.073642 1993 raft_consensus.cc:2396] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260924 00:22:34.094514 4402 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.096458 4456 raft_consensus.cc:1067] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
W20260924 00:22:34.097553 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 54056.6us
I20260924 00:22:34.098868 4423 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:34.099472 4438 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.101413 4434 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.101557 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.110481 1990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:34.110908 2137 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:34.111169 1990 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:34.111591 1990 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:34.112532 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:34.104032 4354 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:34.106635 4464 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:34.123915 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:34.124909 1992 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:34.130375 2036 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.110212 4421 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.139541 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:34.143234 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:34.139285 2137 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:34.138965 4438 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:34.145752 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:34.136198 4403 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:34.150633 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.163861 2144 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:34.151451 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.152103 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.155141 2066 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:34.165277 2138 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:34.161998 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:34.168751 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.170323 4421 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:34.170820 4421 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:34.171315 4421 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:34.171787 1990 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.138837 4250 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:34.179579 1990 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:34.180635 4433 raft_consensus.cc:979] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:34.183586 4250 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:34.200529 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:34.194347 4400 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:34.206327 2109 consensus_peers.cc:487] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4917c5d11b34f7aaa5cbdcf98709f22. This is attempt 1: this message will repeat every 5th retry.
W20260924 00:22:34.216914 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 87921.9us
I20260924 00:22:34.182542 4438 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.219151 4250 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:34.221055 2641 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:34.221851 4455 raft_consensus.cc:1067] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:34.222784 4463 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:34.223255 4463 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.182155 4403 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.225597 4400 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:34.227137 4403 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:34.229107 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:34.229918 2640 raft_consensus.cc:2379] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:34.228691 2641 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
W20260924 00:22:34.245064 4473 log.cc:924] Time spent T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20260924 00:22:34.245378 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 46865us
I20260924 00:22:34.228845 4421 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.248818 4421 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.244012 4438 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.252182 1960 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:34.253428 4430 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:34.253888 4430 raft_consensus.cc:686] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.255810 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.256028 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:34.256392 2062 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:34.257416 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:34.259311 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:34.257359 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:34.254621 4430 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.277635 4463 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:34.279881 1890 catalog_manager.cc:5495] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 8, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:34.281920 1991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:34.282665 1991 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:34.283094 1991 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:34.286675 1960 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:34.287392 2062 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:34.284793 1890 catalog_manager.cc:5495] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 5 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:34.278632 4286 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:34.273902 4432 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:34.302503 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.302698 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.303463 2140 raft_consensus.cc:3032] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:34.304113 2140 raft_consensus.cc:729] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:34.304756 4286 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:34.303509 2143 raft_consensus.cc:3037] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:34.314656 4286 raft_consensus.cc:686] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.316004 4286 consensus_queue.cc:237] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.317700 1890 catalog_manager.cc:5495] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:34.324465 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.325440 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.325774 2142 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:34.326040 2136 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:34.329937 4147 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:34.330428 4147 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.332295 4147 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:34.298151 4432 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:34.333491 4432 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:34.333292 4452 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:34.340108 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.300422 4292 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:34.340732 2137 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:34.340972 4292 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.343273 4292 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.278191 4250 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.347383 1991 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.347836 4250 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.349085 1991 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.351795 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:34.352348 2062 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:34.305038 2140 consensus_queue.cc:260] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.354746 2067 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 9: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:34.355684 2143 raft_consensus.cc:2454] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:34.320737 1890 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 8, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:34.359854 2140 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:34.364761 4403 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:34.365669 4403 raft_consensus.cc:3032] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:34.365967 4403 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:34.366698 4403 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.367841 4403 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:34.367981 2140 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:34.372246 2109 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:34.372668 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:34.373111 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:34.373720 2062 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:34.373374 2640 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
W20260924 00:22:34.377614 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 75407.2us
I20260924 00:22:34.381839 1960 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.385532 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33232 (request call id 218) took 1995 ms. Trace:
I20260924 00:22:34.386941 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.388435 4483 raft_consensus.cc:2781] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:34.387921 1991 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:34.391047 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:34.391458 4483 raft_consensus.cc:686] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.392330 4483 consensus_queue.cc:237] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.396701 4462 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:34.397150 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.397737 2067 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:34.402647 2109 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:34.406204 4463 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:34.406483 2067 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:34.408272 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.408980 2140 raft_consensus.cc:3032] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:34.409345 2140 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:34.410094 2140 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.411218 4463 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:34.411779 4463 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:34.379459 2035 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:34.411468 2140 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:34.414400 4250 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:34.424793 2140 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:34.435148 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.445475 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:34.385937 1893 rpcz_store.cc:276] 0924 00:22:32.389742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:32.393261 (+ 3519us) service_pool.cc:227] Handling call
0924 00:22:34.385473 (+1992212us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:34.045639 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:34.046111 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:34.046148 (+ 37us) write_op.cc:435] Acquired schema lock
0924 00:22:34.046166 (+ 18us) tablet.cc:660] Decoding operations
0924 00:22:34.047072 (+ 906us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:34.047124 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:34.047151 (+ 27us) tablet.cc:683] Acquiring locks for 8 operations
0924 00:22:34.047928 (+ 777us) tablet.cc:699] Row locks acquired
0924 00:22:34.047948 (+ 20us) write_op.cc:260] PREPARE: finished
0924 00:22:34.048083 (+ 135us) write_op.cc:270] Start()
0924 00:22:34.048187 (+ 104us) write_op.cc:276] Timestamp: P: 1790209354048040 usec, L: 0
0924 00:22:34.048206 (+ 19us) op_driver.cc:331] REPLICATION: starting
0924 00:22:34.048891 (+ 685us) log.cc:842] Serialized 3206 byte log entry
0924 00:22:34.050820 (+ 1929us) op_driver.cc:444] REPLICATION: finished
0924 00:22:34.051250 (+ 430us) write_op.cc:301] APPLY: starting
0924 00:22:34.051333 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:34.051559 (+ 226us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:34.051672 (+ 113us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:34.052492 (+ 820us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:34.052711 (+ 219us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0924 00:22:34.052761 (+ 50us) write_op.cc:312] APPLY: finished
0924 00:22:34.053002 (+ 241us) log.cc:842] Serialized 65 byte log entry
0924 00:22:34.053180 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:34.053299 (+ 119us) write_op.cc:454] Released schema lock
0924 00:22:34.053349 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:34.053612 (+ 263us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:34.053636 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34533,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":216,"apply.run_cpu_time_us":2666,"apply.run_wall_time_us":2666,"num_ops":8,"prepare.queue_time_us":4067,"prepare.run_cpu_time_us":3512,"prepare.run_wall_time_us":3512,"replication_time_us":2523}]]}
I20260924 00:22:34.445112 4193 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:34.444253 4436 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:34.458123 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:34.458808 2140 raft_consensus.cc:595] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:34.459080 2035 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:34.463102 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:34.454203 4421 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:34.473340 1960 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:34.459941 4193 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:34.467943 4315 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:34.485903 4193 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:34.484099 4292 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:34.462671 4436 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.489398 4315 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:34.544641 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.545480 1993 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:34.545913 2140 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:34.530161 4315 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:34.564314 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 86671.7us
I20260924 00:22:34.570320 2140 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:22:34.578347 2035 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:34.588289 1942 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 8c129c01e1c047628f6a87b001aa7258 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:34.597808 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2730c02ac0a4812841172254e284b90"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:34.600667 4463 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.601295 2140 raft_consensus.cc:595] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:34.604606 2145 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:34.611797 4489 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:34.612360 4489 raft_consensus.cc:686] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:34.614089 4436 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:34.622838 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 47c6ea1617904bf9bf2b2d904d0e2ae5, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:34.624119 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 47c6ea1617904bf9bf2b2d904d0e2ae5, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:34.632176 4486 log.cc:924] Time spent T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260924 00:22:34.638792 4489 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.643100 4463 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:34.675980 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bd66a46f82114470b22385448fd57d51, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:34.676654 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet bd66a46f82114470b22385448fd57d51, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.682102 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.682741 2068 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:34.673697 4432 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.687867 1996 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 9: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:34.690618 4432 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.691749 2068 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:34.701884 4315 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.704042 4315 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:34.709640 2109 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.574234 4421 raft_consensus.cc:686] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.713997 4421 consensus_queue.cc:237] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.715828 4454 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:34.716565 4454 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:22:34.720933 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 115228us
W20260924 00:22:34.726605 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 148329us
I20260924 00:22:34.729791 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:34.729931 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:34.730751 2640 raft_consensus.cc:2379] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:34.717886 4454 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.732247 2071 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:34.735900 2640 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:34.735181 2071 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:34.737840 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:22:34.739074 4432 consensus_queue.cc:1046] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:34.739421 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:34.742084 4438 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:34.744288 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:34.742996 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.745607 2143 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 9: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:34.750229 4432 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:34.766984 4432 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.753646 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:34.779973 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bf0af13699974118a86f602c1fa9490b, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
W20260924 00:22:34.780376 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet bf0af13699974118a86f602c1fa9490b, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:34.768008 4432 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.782291 4432 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:34.757289 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:34.710700 4193 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.751209 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.788550 2143 raft_consensus.cc:2354] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:34.789988 4363 raft_consensus.cc:979] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:34.790478 4363 raft_consensus.cc:1067] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:34.799095 2035 leader_election.cc:398] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:34.799458 2035 leader_election.cc:401] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260924 00:22:34.799997 4489 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:34.804095 1993 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2730c02ac0a4812841172254e284b90"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:34.804814 1993 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:34.805222 1993 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:34.812387 4489 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6.
I20260924 00:22:34.817066 4193 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:34.822148 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.823210 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:34.823637 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.823851 2649 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:34.826320 1993 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.827971 1993 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.830210 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.830833 2144 raft_consensus.cc:3032] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:34.831096 2144 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:34.831458 4495 ts_tablet_manager.cc:927] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.831784 2144 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.832933 2144 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:34.834174 2068 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260924 00:22:34.838243 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.838749 2068 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:34.843698 2062 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260924 00:22:34.844475 2145 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260924 00:22:34.844568 2062 pending_rounds.cc:76] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 3
I20260924 00:22:34.845053 2062 pending_rounds.cc:97] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260924 00:22:34.845459 2062 raft_consensus.cc:2866] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:34.848896 4147 raft_consensus.cc:979] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:34.849445 4147 raft_consensus.cc:1067] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:34.850263 4495 tablet_copy_client.cc:316] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:22:34.854436 1988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:34.854748 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:34.854983 1988 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:34.855446 1988 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:34.855717 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet a4917c5d11b34f7aaa5cbdcf98709f22 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:22:34.856061 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:34.856330 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet a4917c5d11b34f7aaa5cbdcf98709f22 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-a4917c5d11b34f7aaa5cbdcf98709f22
I20260924 00:22:34.858548 4436 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:34.858973 4490 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260924 00:22:34.852097 1890 catalog_manager.cc:5495] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index -1 to 4, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:34.859498 2067 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:34.860529 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.861240 4454 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:34.847867 2144 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:34.861713 4454 raft_consensus.cc:686] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:34.862838 4483 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:34.862545 4454 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:34.863497 4483 consensus_queue.cc:1225] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 948356bf66a941ef8249ecbd0ec94f6d log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260924 00:22:34.863626 4490 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:34.864383 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:34.864866 4286 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:34.865264 4292 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:34.865752 4292 raft_consensus.cc:686] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.866578 4292 consensus_queue.cc:237] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.866799 1890 catalog_manager.cc:5495] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 9, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:34.869975 1890 catalog_manager.cc:5495] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:34.871672 2155 tablet_copy_source_session.cc:205] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:34.877031 1890 catalog_manager.cc:5495] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:34.879786 2062 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:34.879922 1890 catalog_manager.cc:5495] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 8, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 8 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:34.886504 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4917c5d11b34f7aaa5cbdcf98709f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:34.888571 1890 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 8 to 9, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:34.890791 1890 catalog_manager.cc:5495] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:34.891990 2143 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260924 00:22:34.886060 2649 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:34.894975 4421 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:34.904938 2067 pending_rounds.cc:76] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 3
I20260924 00:22:34.846282 2068 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:34.905303 2067 pending_rounds.cc:97] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260924 00:22:34.904529 1890 catalog_manager.cc:5495] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 9, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:34.905748 2067 raft_consensus.cc:2866] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:34.913985 2143 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:34.917704 4434 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:34.912344 1988 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.921512 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:34.922106 2069 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:34.913542 4421 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.017s
I20260924 00:22:34.935416 2069 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:34.936501 1960 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:34.936614 1988 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:34.952632 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:34.953491 2140 raft_consensus.cc:3032] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:34.953843 2140 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:34.954730 2140 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.959546 2140 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:34.947382 4483 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:34.972340 2140 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:34.978222 4483 raft_consensus.cc:686] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:34.981559 4483 consensus_queue.cc:237] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:34.987991 2137 raft_consensus.cc:1261] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:34.996524 4436 consensus_queue.cc:1035] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:35.050676 4495 tablet_copy_client.cc:791] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:35.117147 4495 tablet_copy_client.cc:655] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:35.171564 4495 tablet_copy_client.cc:531] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:35.194299 4495 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:35.202039 1993 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260924 00:22:35.202462 1993 pending_rounds.cc:76] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 3
I20260924 00:22:35.202675 1993 pending_rounds.cc:97] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260924 00:22:35.202885 1993 raft_consensus.cc:2866] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:35.205114 4354 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:35.214707 2068 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:35.222628 4103 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260924 00:22:35.230652 2143 raft_consensus.cc:1261] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:35.231870 2144 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260924 00:22:35.233505 1991 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260924 00:22:35.233489 4434 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:35.235502 4489 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260924 00:22:35.240995 4492 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20260924 00:22:35.371575 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 82407.8us
I20260924 00:22:35.392047 2068 raft_consensus.cc:1261] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:35.417207 2071 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:35.396821 4421 consensus_queue.cc:1035] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260924 00:22:35.416781 2137 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:35.425359 4436 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:35.428905 2062 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:35.431306 4421 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:35.453264 1891 catalog_manager.cc:5495] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:35.455534 2137 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:35.457257 4421 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:35.463119 1891 catalog_manager.cc:5495] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 7, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:35.465659 4147 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:35.471297 2649 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:35.464754 1891 catalog_manager.cc:5495] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:35.489609 1891 catalog_manager.cc:5495] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 7 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:35.493949 1891 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:35.523906 4523 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260924 00:22:35.536109 2071 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:35.540427 4434 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:35.595207 4495 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:22:35.596108 4495 tablet_bootstrap.cc:492] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:35.596987 4495 ts_tablet_manager.cc:1397] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.403s user 0.014s sys 0.021s
I20260924 00:22:35.675031 4495 raft_consensus.cc:348] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:35.676031 4495 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:22:35.676685 4495 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:35.698971 1890 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:35.705204 4495 ts_tablet_manager.cc:1428] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.104s user 0.006s sys 0.002s
I20260924 00:22:35.726848 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-a4917c5d11b34f7aaa5cbdcf98709f22 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:22:35.727269 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-a4917c5d11b34f7aaa5cbdcf98709f22 on tablet a4917c5d11b34f7aaa5cbdcf98709f22 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:35.970974 2649 raft_consensus.cc:1203] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Deduplicated request from leader. Original: 5.3->[5.4-5.4] Dedup: 5.4->[]
I20260924 00:22:35.993542 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:36.095149 2137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "150b87da8b7e456cb68c4549975535df"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:36.095978 2137 raft_consensus.cc:595] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:36.104769 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:36.105415 2145 raft_consensus.cc:595] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:36.116297 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:36.116914 2143 raft_consensus.cc:595] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:36.119789 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:36.120344 2143 raft_consensus.cc:595] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:36.123643 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:36.124126 2143 raft_consensus.cc:595] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:36.128077 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:22:36.129739 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:36.130330 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:36.139365 4280 raft_consensus.cc:979] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:36.139955 4280 raft_consensus.cc:1067] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:36.140491 4523 raft_consensus.cc:979] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:36.140928 4523 raft_consensus.cc:1067] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:36.141732 1988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "150b87da8b7e456cb68c4549975535df"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:36.142259 1988 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:36.142565 2069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:36.143075 2069 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:36.143406 2069 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:36.142580 1988 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:36.150068 2069 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.157465 2069 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:36.157799 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:36.158402 1991 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:36.158006 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:36.160576 2137 raft_consensus.cc:3032] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:36.160876 2137 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:36.162235 1988 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.163796 1991 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:36.165056 2036 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:36.165868 4520 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:36.166371 4520 raft_consensus.cc:686] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:36.167176 4520 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.169646 1988 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:36.175058 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:36.175603 2143 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:36.175875 2143 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:36.176537 2143 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.177758 2143 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:36.183634 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:36.184230 2069 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:36.199558 2143 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:36.191216 2069 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:36.201323 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:36.161566 2137 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.207679 2137 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:36.221251 2137 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:36.211475 4436 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:36.238925 4436 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:36.248698 1892 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 7 to 8, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 8 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:36.262580 4436 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.318246 4280 raft_consensus.cc:979] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:36.318598 4523 raft_consensus.cc:979] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:36.318732 4280 raft_consensus.cc:1067] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:36.319226 4523 raft_consensus.cc:1067] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:36.322495 2062 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:36.323060 2062 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:36.323549 2062 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:36.333115 2069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:36.332090 2062 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:36.333720 2069 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:36.334084 2069 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:36.336234 2062 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:36.338920 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:36.340919 2649 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:36.340976 2069 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.347091 2069 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:36.347764 2649 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:36.348888 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:36.349689 4520 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:36.350153 4520 raft_consensus.cc:686] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:36.350950 4520 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:36.355023 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:36.355618 1988 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:36.361014 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:36.361657 2137 raft_consensus.cc:3032] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:36.361924 2137 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:36.362582 2137 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:36.363659 2137 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:36.368641 2137 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:36.370602 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:36.371094 2137 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:36.371321 2137 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:36.371922 2137 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.372963 2137 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:36.374010 1988 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:36.375576 2036 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:36.379521 4541 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:36.382074 4541 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:36.383700 4541 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.393486 2137 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:36.404089 1893 catalog_manager.cc:5495] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:36.424480 1892 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 4 to 5, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 5 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:36.426482 1892 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 6 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:36.497838 4540 raft_consensus.cc:1050] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:22:36.504490 4280 raft_consensus.cc:979] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:36.503897 4540 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 2, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:36.505066 4280 raft_consensus.cc:1067] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:36.507745 1991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:36.508245 1991 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:36.508535 1991 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:36.529752 1991 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.532696 2649 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:22:36.533386 1988 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:22:36.535302 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:36.535938 2069 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:36.537343 4540 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260924 00:22:36.541364 4539 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:36.544265 2062 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:22:36.551258 1991 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:36.552515 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:36.553126 2137 raft_consensus.cc:3032] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:36.553462 2137 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:36.553960 4540 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:36.554195 2137 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.555467 2137 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:36.559952 2069 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:36.560984 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:36.562649 2137 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:36.564070 4436 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:36.564754 4436 raft_consensus.cc:686] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:36.565747 4436 consensus_queue.cc:237] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:36.600406 1991 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:36.602468 4541 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:36.609550 4456 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:36.633327 2062 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:36.642767 1988 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:36.645104 1892 catalog_manager.cc:5495] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:36.671501 1892 catalog_manager.cc:5495] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 4 to 5, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:36.685767 2137 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:36.704603 2649 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 4 to 5, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:36.705971 4537 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:36.755012 2145 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:36.758111 4436 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:36.824551 2145 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:36.838207 2649 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:36.845324 2069 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:36.847188 4436 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:36.862010 2062 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:36.880260 1993 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:36.884979 4520 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:36.884287 4280 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:36.888422 4258 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:36.892004 2145 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:36.909230 4520 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:36.916097 2649 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:36.918511 4541 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:36.924582 4523 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:36.941507 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a4917c5d11b34f7aaa5cbdcf98709f22 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:22:36.957284 2071 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:36.966713 2069 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:36.968327 4434 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:36.992331 2145 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:36.994299 1892 catalog_manager.cc:5495] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 5 to 6, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:37.006881 2649 raft_consensus.cc:2932] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:37.011641 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet a4917c5d11b34f7aaa5cbdcf98709f22 with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55036
W20260924 00:22:37.043911 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 56277.8us
I20260924 00:22:37.040529 4537 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260924 00:22:37.133891 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 78707.2us
I20260924 00:22:37.155068 4570 tablet_replica.cc:330] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:22:37.156101 4570 raft_consensus.cc:2229] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:37.156876 4570 raft_consensus.cc:2258] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Raft consensus is shut down!
W20260924 00:22:37.157610 4568 log.cc:924] Time spent T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
W20260924 00:22:37.199350 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 44182.9us
W20260924 00:22:37.200336 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 51630.5us
I20260924 00:22:37.210232 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:37.218580 2137 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260924 00:22:37.310866 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 108726us
W20260924 00:22:37.322383 4568 log.cc:924] Time spent T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
W20260924 00:22:37.358479 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 153976us
I20260924 00:22:37.413899 4570 ts_tablet_manager.cc:1905] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:22:37.391021 4582 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
W20260924 00:22:37.499253 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 125438us
W20260924 00:22:37.528573 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 76134.2us
W20260924 00:22:37.560420 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 44511us
I20260924 00:22:37.654556 4570 ts_tablet_manager.cc:1918] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.5
I20260924 00:22:37.659087 4570 log.cc:1196] T a4917c5d11b34f7aaa5cbdcf98709f22 P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/a4917c5d11b34f7aaa5cbdcf98709f22
I20260924 00:22:37.707023 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet a4917c5d11b34f7aaa5cbdcf98709f22 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:22:37.716043 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:37.717265 2145 raft_consensus.cc:595] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:37.852934 4597 raft_consensus.cc:979] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:37.853524 4597 raft_consensus.cc:1067] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:37.873942 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85bede790e49439d97e8031810eee902"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:37.874606 2140 raft_consensus.cc:595] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:37.881402 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba683df200644d888896fb84dbc748e8"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:37.882040 2145 raft_consensus.cc:595] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:37.882061 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:37.883708 2068 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:37.884172 2068 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:37.898826 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:37.899541 2145 raft_consensus.cc:595] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:37.911586 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:37.912312 2145 raft_consensus.cc:595] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:37.915302 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:22:37.919153 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:37.919948 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260924 00:22:37.921157 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 48687.1us
I20260924 00:22:37.941661 4597 raft_consensus.cc:979] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:37.942188 4597 raft_consensus.cc:1067] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:37.944137 1996 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "85bede790e49439d97e8031810eee902"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:37.944834 1996 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:37.945246 1996 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:37.965108 4258 raft_consensus.cc:979] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:37.966717 4258 raft_consensus.cc:1067] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:37.975121 1987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:37.975759 1987 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:37.976130 1987 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:22:37.994429 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 57800.1us
I20260924 00:22:38.014756 2068 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.016963 1996 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.023630 4586 raft_consensus.cc:979] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:38.037531 4586 raft_consensus.cc:1067] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:38.039314 1991 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba683df200644d888896fb84dbc748e8"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:38.039870 1991 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:38.040176 1991 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:38.042137 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:38.042899 2062 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:38.045660 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:38.048743 1988 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:38.053373 2068 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:38.059418 1988 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:38.029238 1987 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.063158 1996 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:38.062654 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:38.065282 2145 raft_consensus.cc:3032] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:38.067353 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:38.067253 1991 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.070977 2140 raft_consensus.cc:3032] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:38.069243 2145 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:38.073654 1987 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:38.071353 2140 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:38.081112 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:38.081883 2068 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:38.083876 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:38.084079 2036 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:38.084496 2143 raft_consensus.cc:3032] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:38.084105 2140 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.084793 2143 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:38.085045 4567 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:38.085569 4567 raft_consensus.cc:686] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:38.085705 2143 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.086839 2143 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:38.086473 4567 consensus_queue.cc:237] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.083138 2145 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.118574 1991 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:38.120359 2140 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:38.123572 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:38.124186 2069 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:38.127930 2145 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:38.136264 2062 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:38.151219 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:38.162009 2143 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:38.191640 4606 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:38.224175 4606 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:38.273758 4606 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.258773 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:38.286057 2144 raft_consensus.cc:3032] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:38.286429 2144 raft_consensus.cc:729] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:38.287209 2144 consensus_queue.cc:260] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.288619 2144 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:38.293507 4573 raft_consensus.cc:979] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:38.294063 4573 raft_consensus.cc:1067] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:38.309425 2071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:38.310025 2071 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:38.310376 2071 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:38.317255 2069 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:38.319113 2068 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:38.329321 2140 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:38.330911 2145 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:38.336033 2071 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:38.337931 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 113456us
I20260924 00:22:38.374539 2071 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:38.351316 2144 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:38.357477 1890 catalog_manager.cc:5495] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:38.384347 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:38.385181 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:38.389796 1991 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:22:38.396217 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 179360us
I20260924 00:22:38.397046 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:38.398491 2144 raft_consensus.cc:3032] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:38.399818 4589 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:38.401240 2144 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:38.401670 4589 raft_consensus.cc:686] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:38.410138 4589 consensus_queue.cc:237] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.428786 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:38.409734 2144 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.435707 2144 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:38.438863 4617 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:38.438580 1991 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:38.444556 4617 raft_consensus.cc:686] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:38.445732 4617 consensus_queue.cc:237] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.484654 2144 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:38.495003 2036 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:38.511236 4579 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:38.511991 4579 raft_consensus.cc:686] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:38.513216 4579 consensus_queue.cc:237] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:38.530274 1892 catalog_manager.cc:5495] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:38.541594 2144 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:38.578506 1890 catalog_manager.cc:5495] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:38.579515 1996 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:38.580346 1890 catalog_manager.cc:5495] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 5, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 5 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:38.567402 1892 catalog_manager.cc:5495] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 4 to 5, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 5 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:38.586719 4575 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:38.597326 2069 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:22:38.612247 4572 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260924 00:22:38.615302 2109 consensus_peers.cc:487] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 708537d593c14817accda67f9fd081b4. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:38.638985 1987 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260924 00:22:38.655419 4597 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:38.669456 2071 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:38.675308 4589 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:38.695365 2069 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:38.692124 4539 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:38.747638 2144 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:38.755451 2143 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:38.760341 4579 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260924 00:22:38.764030 1890 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:38.779359 4617 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:38.796958 2069 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:22:38.898979 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 82849.1us
I20260924 00:22:38.946352 2067 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260924 00:22:38.954248 4631 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.131s user 0.001s sys 0.001s
W20260924 00:22:39.035058 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 109555us
I20260924 00:22:39.039757 2144 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:38.988480 4617 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260924 00:22:39.050446 2144 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:39.046489 1988 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:39.054692 4603 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260924 00:22:39.054733 4567 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260924 00:22:39.164588 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 211360us
I20260924 00:22:39.203389 2070 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:39.216375 4651 ts_tablet_manager.cc:927] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:39.219925 2143 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260924 00:22:39.222950 2035 consensus_peers.cc:487] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54b1b238378448a5987c49aafc3b6313. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:39.267488 4637 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:39.288869 1996 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:39.290637 1996 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:39.326645 4575 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:39.326954 4603 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:39.359122 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:39.361003 4651 tablet_copy_client.cc:316] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:22:39.393102 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet 708537d593c14817accda67f9fd081b4 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:22:39.393903 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet 708537d593c14817accda67f9fd081b4 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-708537d593c14817accda67f9fd081b4
I20260924 00:22:39.414533 4648 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:39.394596 4606 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260924 00:22:39.432777 2144 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:39.561955 2155 tablet_copy_source_session.cc:205] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:39.567255 2145 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260924 00:22:39.583621 4656 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
W20260924 00:22:39.601799 4653 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260924 00:22:39.605505 4651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708537d593c14817accda67f9fd081b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:39.621685 4589 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:39.630206 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "150b87da8b7e456cb68c4549975535df"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:39.636142 1996 raft_consensus.cc:595] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:39.664062 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:39.690649 2145 raft_consensus.cc:595] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:39.707414 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:39.710316 2145 raft_consensus.cc:595] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:39.807981 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 62231us
I20260924 00:22:39.817585 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:39.828109 2145 raft_consensus.cc:595] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:39.856263 4658 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.137s user 0.001s sys 0.001s
I20260924 00:22:39.961840 4616 raft_consensus.cc:979] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:39.968613 4616 raft_consensus.cc:1067] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:39.982403 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:39.984129 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:39.984809 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:40.032531 4666 ts_tablet_manager.cc:927] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:40.044660 4666 tablet_copy_client.cc:316] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:22:40.057193 4651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45582 (request call id 409) took 1060 ms. Trace:
I20260924 00:22:40.057679 4651 rpcz_store.cc:276] 0924 00:22:38.996738 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:38.996900 (+ 162us) service_pool.cc:227] Handling call
0924 00:22:39.216775 (+219875us) ts_tablet_manager.cc:928] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
0924 00:22:39.573083 (+356308us) tablet_copy_client.cc:361] Tablet copy session begun
0924 00:22:39.828388 (+255305us) tablet_metadata.cc:641] Metadata flushed
0924 00:22:39.900663 (+ 72275us) tablet_copy_client.cc:490] Wrote new tablet metadata
0924 00:22:40.056386 (+155723us) tablet_copy_client.cc:411] Wrote new consensus metadata
0924 00:22:40.056679 (+ 293us) tablet_replica.cc:205] Creating consensus instance
0924 00:22:40.057035 (+ 356us) ts_tablet_manager.cc:983] Replica registered.
0924 00:22:40.057098 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":257,"tablet-copy.queue_time_us":218880,"thread_start_us":194300,"threads_started":2}
I20260924 00:22:40.059278 4651 tablet_copy_client.cc:791] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:40.059870 4651 tablet_copy_client.cc:655] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:40.064730 1995 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:40.075754 4651 tablet_copy_client.cc:531] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:40.084860 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 54b1b238378448a5987c49aafc3b6313 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:22:40.085515 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 54b1b238378448a5987c49aafc3b6313 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-54b1b238378448a5987c49aafc3b6313
W20260924 00:22:40.088930 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 226246us
I20260924 00:22:40.106320 1892 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:40.113881 4539 raft_consensus.cc:979] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:40.114643 4640 raft_consensus.cc:1067] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:40.121501 2081 tablet_copy_source_session.cc:205] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:40.158538 4651 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:40.159317 4666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b1b238378448a5987c49aafc3b6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:40.176395 2062 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "150b87da8b7e456cb68c4549975535df"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:22:40.176976 2062 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:40.177474 2062 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.190353 2062 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:40.203605 4640 raft_consensus.cc:979] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:40.204157 4640 raft_consensus.cc:1067] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:40.205746 2144 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:40.207119 1988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:40.207683 1988 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:40.207995 1988 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:40.208621 2067 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:40.222031 1988 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:40.224017 2067 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:40.226291 2062 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:40.227738 1988 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:40.229437 4644 raft_consensus.cc:979] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:40.235509 4644 raft_consensus.cc:1067] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:40.242887 2067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:40.243438 2067 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:40.243754 2067 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.245185 1995 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:40.245728 1995 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:40.246073 1995 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.251151 2067 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:40.254809 2067 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:22:40.271541 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:40.271440 4646 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:40.272174 2144 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.275849 4666 tablet_copy_client.cc:791] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:40.276441 4666 tablet_copy_client.cc:655] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:40.277199 1995 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:40.281092 2144 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:40.301299 1995 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:40.309645 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:40.310372 2138 raft_consensus.cc:3032] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:40.310698 2138 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:40.323222 2138 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:40.324862 2138 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.326586 4666 tablet_copy_client.cc:531] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:40.343533 2138 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:40.347820 1989 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:40.349602 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:40.349879 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:40.350188 1996 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:40.350433 1987 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.350586 1996 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:40.351377 1996 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:40.352447 1996 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.358843 1987 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:40.360208 1996 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:40.371068 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:40.371624 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:40.371752 2144 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:40.372182 2140 raft_consensus.cc:3032] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:40.372280 2144 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:40.372571 2140 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:40.374341 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:40.374089 2140 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:40.374876 2067 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:40.379346 2035 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:40.386163 2144 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:40.388916 2140 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.390170 4603 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:40.390751 4603 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:40.391732 4603 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:40.394137 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:40.394917 2649 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:40.396944 2035 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:40.386023 1960 consensus_peers.cc:487] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47c6ea1617904bf9bf2b2d904d0e2ae5. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:40.406322 2144 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:40.408483 4567 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:40.416093 4567 raft_consensus.cc:686] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:40.408531 4646 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260924 00:22:40.417104 4567 consensus_queue.cc:237] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:40.417790 2144 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:40.457526 2649 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
W20260924 00:22:40.470074 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 60820.6us
I20260924 00:22:40.469715 2067 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:40.481029 2140 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:40.466612 4666 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:40.485764 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:40.507757 1960 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:40.504317 4616 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:40.522089 4637 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:40.536767 4637 raft_consensus.cc:686] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:40.537902 4637 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:40.542538 4616 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:40.544095 4651 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:22:40.543618 4616 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:40.549160 4651 tablet_bootstrap.cc:492] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:40.550012 4651 ts_tablet_manager.cc:1397] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.392s user 0.023s sys 0.026s
W20260924 00:22:40.571249 4681 log.cc:924] Time spent T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260924 00:22:40.583375 1890 catalog_manager.cc:5495] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 6 to 7, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:40.600030 4589 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:40.604560 2139 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
W20260924 00:22:40.614027 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 96787.2us
I20260924 00:22:40.658582 1893 catalog_manager.cc:5495] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 7 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260924 00:22:40.797923 4687 log.cc:924] Time spent T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260924 00:22:40.804062 4651 raft_consensus.cc:348] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:40.805084 4651 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
W20260924 00:22:40.812081 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 128654us
I20260924 00:22:40.818830 4690 ts_tablet_manager.cc:927] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:22:40.810732 4651 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:40.862869 4651 ts_tablet_manager.cc:1428] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.307s user 0.014s sys 0.009s
I20260924 00:22:40.865968 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-708537d593c14817accda67f9fd081b4 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:22:40.866380 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-708537d593c14817accda67f9fd081b4 on tablet 708537d593c14817accda67f9fd081b4 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:40.904816 4690 tablet_copy_client.cc:316] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.1:41775
W20260924 00:22:40.936184 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 57479.9us
I20260924 00:22:40.924889 4589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:42922 (request call id 7) took 1080 ms. Trace:
I20260924 00:22:40.941788 4589 rpcz_store.cc:276] 0924 00:22:39.844446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:39.844592 (+ 146us) service_pool.cc:227] Handling call
0924 00:22:40.062219 (+217627us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b54003b0c20. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4a51dc0 00007fb6d4ac2bd2 00007fb6d4ac1890 00007fb6d99fef39 00007fb6d30ea704 00007fb6d30ea692 00007fb6d30ea5c1 00007fb6d30ea53c 00007fb6d30e97f2 00007fb6d2ee694c
0924 00:22:40.078690 (+ 16471us) log.cc:842] Serialized 492 byte log entry
0924 00:22:40.924773 (+846083us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":7104,"mutex_wait_us":26462,"raft.queue_time_us":31034,"raft.run_cpu_time_us":1230,"raft.run_wall_time_us":1230,"spinlock_wait_cycles":27988352,"thread_start_us":13099,"threads_started":1,"wal-append.queue_time_us":11663}
I20260924 00:22:40.993861 1892 catalog_manager.cc:5495] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 6 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } }
W20260924 00:22:41.011349 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 60553.3us
I20260924 00:22:41.051463 2138 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260924 00:22:41.071888 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 144161us
I20260924 00:22:41.092926 2138 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260924 00:22:41.095237 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 164964us
I20260924 00:22:41.100013 2006 tablet_copy_service.cc:140] P 3eb374c9744b4c7e92dc749aece177dd: Received BeginTabletCopySession request for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:42938)
I20260924 00:22:41.100800 2006 tablet_copy_service.cc:161] P 3eb374c9744b4c7e92dc749aece177dd: Beginning new tablet copy session on tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:42938: session id = 9e79063532384ada8ea8f8205478adcf-47c6ea1617904bf9bf2b2d904d0e2ae5
W20260924 00:22:41.113654 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 175716us
I20260924 00:22:41.120311 2006 tablet_copy_source_session.cc:205] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:41.119412 2070 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
W20260924 00:22:41.154052 1942 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 8c129c01e1c047628f6a87b001aa7258 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
W20260924 00:22:41.158780 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 98489.7us
W20260924 00:22:41.159621 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 86837us
I20260924 00:22:41.167937 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:41.180855 4690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:41.192229 4666 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:22:41.193717 4666 tablet_bootstrap.cc:492] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:41.194527 4666 ts_tablet_manager.cc:1397] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.755s user 0.088s sys 0.076s
I20260924 00:22:41.197664 4666 raft_consensus.cc:348] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:41.198712 4666 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:22:41.199407 4666 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:41.202109 4666 ts_tablet_manager.cc:1428] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 00:22:41.204048 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-54b1b238378448a5987c49aafc3b6313 received from {username='slave'} at 127.0.0.1:43194
I20260924 00:22:41.204421 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-54b1b238378448a5987c49aafc3b6313 on tablet 54b1b238378448a5987c49aafc3b6313 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:41.203231 2139 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:41.211826 4686 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:41.217657 1891 catalog_manager.cc:5495] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:41.228209 2063 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:41.237648 2144 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:41.238204 4664 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260924 00:22:41.282665 1987 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:41.284710 2641 raft_consensus.cc:1203] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
I20260924 00:22:41.285430 1987 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:22:41.293350 4690 tablet_copy_client.cc:791] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:41.294076 4690 tablet_copy_client.cc:655] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:41.313189 4690 tablet_copy_client.cc:531] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:41.315536 2641 raft_consensus.cc:1261] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:41.326257 4664 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:41.325791 1891 catalog_manager.cc:5495] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 8 to 9, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:41.326531 4648 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:41.344739 4690 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:41.349141 2641 raft_consensus.cc:1203] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 LEARNER]: Deduplicated request from leader. Original: 6.4->[6.5-6.5] Dedup: 6.5->[]
I20260924 00:22:41.350312 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:41.350871 1987 raft_consensus.cc:595] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:41.353816 2070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:41.354329 2070 raft_consensus.cc:595] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:41.368909 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:41.369730 2138 raft_consensus.cc:595] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:41.374249 2070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:41.374816 2070 raft_consensus.cc:595] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:41.377964 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:41.379230 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:41.379694 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:41.375862 4638 raft_consensus.cc:979] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:41.370257 4652 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260924 00:22:41.375645 4616 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260924 00:22:41.390897 4700 raft_consensus.cc:1067] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:41.392853 4645 consensus_queue.cc:1035] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260924 00:22:41.444928 4567 raft_consensus.cc:979] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:41.445523 4655 raft_consensus.cc:1067] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:41.449767 2138 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:60184
I20260924 00:22:41.450376 2138 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:41.457276 2138 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:41.459424 4606 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:41.482473 2140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:57748
I20260924 00:22:41.483039 2140 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:41.483395 2140 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:41.484104 2138 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:41.486995 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:41.487568 2070 raft_consensus.cc:3032] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:41.487838 2070 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:41.488476 2070 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:41.491668 2070 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:41.497640 2138 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:41.501281 2070 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:41.504911 2140 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:41.508908 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:41.509652 1989 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:41.512275 2140 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:41.514928 2109 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:41.517000 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:41.518029 2070 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:41.524427 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:41.525208 1996 raft_consensus.cc:3032] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:41.525548 1996 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:41.527711 1989 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:41.530592 1996 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:41.532011 1996 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:41.554679 2070 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:41.529277 4572 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:41.559918 4572 raft_consensus.cc:686] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:41.560386 2109 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:41.556754 1996 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:41.561635 4258 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:41.561017 4572 consensus_queue.cc:237] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:41.562075 4258 raft_consensus.cc:686] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:41.563030 4258 consensus_queue.cc:237] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:41.609215 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 51435.1us
W20260924 00:22:41.644887 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d1caddf844ef49c0b3664f36bb76b869, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:41.645383 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d1caddf844ef49c0b3664f36bb76b869, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:41.653163 4709 log.cc:924] Time spent T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
W20260924 00:22:41.876827 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 238038us
I20260924 00:22:41.907364 4655 raft_consensus.cc:979] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:41.908217 4664 raft_consensus.cc:1067] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
W20260924 00:22:41.911731 4714 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.247s user 0.002s sys 0.000s
W20260924 00:22:42.064832 4710 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.162s user 0.000s sys 0.001s
W20260924 00:22:42.087080 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 131486us
W20260924 00:22:42.092597 4716 log.cc:924] Time spent T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.240s user 0.002s sys 0.000s
W20260924 00:22:42.127053 4711 log.cc:924] Time spent T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
W20260924 00:22:42.126947 4708 log.cc:924] Time spent T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.338s user 0.000s sys 0.001s
I20260924 00:22:42.186406 4685 raft_consensus.cc:1050] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
W20260924 00:22:42.193887 2034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.000s sys 0.023s
W20260924 00:22:42.194329 2034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.536s user 0.000s sys 0.023s
W20260924 00:22:42.208938 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 44641.4us
W20260924 00:22:42.234632 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.003s sys 0.025s
W20260924 00:22:42.237299 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.611s user 0.003s sys 0.025s
W20260924 00:22:42.237684 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 611255us
I20260924 00:22:42.188977 4685 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 3, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:42.399034 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 126890us
I20260924 00:22:42.414832 4690 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:22:42.415786 4690 tablet_bootstrap.cc:492] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:42.416420 4690 ts_tablet_manager.cc:1397] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 1.072s user 0.029s sys 0.014s
I20260924 00:22:42.445188 1988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:36934
I20260924 00:22:42.445967 1988 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:42.446430 1988 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:42.448681 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:42.516144 4690 raft_consensus.cc:348] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:42.523664 4690 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:22:42.524382 4690 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:42.535465 4701 raft_consensus.cc:1050] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:22:42.540959 1988 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.542719 1988 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:42.562418 1993 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:42.563089 4701 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:42.578933 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:42.591135 2063 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
W20260924 00:22:42.590075 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 66511.6us
I20260924 00:22:42.591447 2063 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:42.594724 1988 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:42.595278 2063 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.599560 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:42.600165 2137 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:42.609109 2138 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260924 00:22:42.616564 2641 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260924 00:22:42.623801 4690 ts_tablet_manager.cc:1428] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.207s user 0.006s sys 0.004s
I20260924 00:22:42.625231 4715 raft_consensus.cc:979] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:42.625779 4715 raft_consensus.cc:1067] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:42.607640 4718 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260924 00:22:42.606098 4644 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260924 00:22:42.632611 4655 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:42.633229 4655 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:42.635021 4752 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:42.640796 4682 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260924 00:22:42.662411 1996 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:42.667991 1996 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:42.668402 1996 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:42.686807 2064 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:42.689333 4664 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:42.689899 4664 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:42.700080 1996 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.700158 2137 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:42.705631 2144 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:42.706401 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:42.717816 1996 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:42.751583 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:42.751793 2069 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:42.752362 2137 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:42.752769 2137 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:42.707165 2006 tablet_copy_service.cc:342] P 3eb374c9744b4c7e92dc749aece177dd: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-47c6ea1617904bf9bf2b2d904d0e2ae5 received from {username='slave'} at 127.0.0.1:42938
I20260924 00:22:42.712050 4606 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:42.780987 4699 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:42.778551 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:42.781418 4664 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:42.786156 2069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:42.785831 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:42.791007 2137 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.803911 2137 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:42.802254 2641 raft_consensus.cc:1203] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Deduplicated request from leader. Original: 9.5->[9.6-9.6] Dedup: 9.6->[]
I20260924 00:22:42.808118 2063 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:42.803433 2069 raft_consensus.cc:595] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:42.823789 4607 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:42.824429 4607 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.826694 4607 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:42.821062 4616 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:42.836303 4648 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:42.837030 4648 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:42.823807 4655 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:42.840322 4648 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:42.833066 4616 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.783682 2006 tablet_copy_service.cc:433] P 3eb374c9744b4c7e92dc749aece177dd: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-47c6ea1617904bf9bf2b2d904d0e2ae5 on tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:42.841382 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:42.842370 4616 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:42.842835 2137 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:42.866707 1891 catalog_manager.cc:5495] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:42.879714 4748 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:42.880333 4748 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:42.882450 4748 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:42.883040 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 75101.7us
I20260924 00:22:42.844854 4606 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:42.893587 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:42.893975 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:42.895064 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:42.895732 1989 raft_consensus.cc:595] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:42.916267 2063 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:42.919478 1891 catalog_manager.cc:5495] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 5 to 6, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 6 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:42.792061 4699 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.939189 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:42.947860 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:42.952425 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:42.956398 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:42.957185 1996 raft_consensus.cc:595] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:42.959221 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:42.948238 4648 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:42.966221 4648 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:42.967411 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:42.968333 4648 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:42.924914 4606 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:42.947983 4695 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:42.973881 4607 raft_consensus.cc:979] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:42.974710 4699 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:42.987694 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:42.977788 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:42.989780 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:42.998808 4682 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:42.997507 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:42.998903 2035 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:42.999625 2066 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:42.971084 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:43.028317 4748 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:43.018254 4695 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:43.045603 4695 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:43.029407 4700 raft_consensus.cc:1067] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:42.999140 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:43.063174 4733 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260924 00:22:43.063374 4640 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260924 00:22:43.068271 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 69440.5us
I20260924 00:22:43.069621 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:43.076156 2064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:43.076828 2064 raft_consensus.cc:595] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:43.077203 2064 raft_consensus.cc:603] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:43.078399 2066 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:43.079859 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:43.088877 1989 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:43.090273 1987 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260924 00:22:43.122455 2136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:57748
I20260924 00:22:43.123272 2136 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:43.123708 2136 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:43.124802 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:43.123656 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:43.131013 4645 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:43.131614 4645 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:43.133725 4645 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:43.071316 4682 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260924 00:22:43.146318 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 50676.4us
I20260924 00:22:43.150370 2649 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:43.154146 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:43.163791 1960 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:43.170329 4664 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.159144 4700 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:43.178810 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:43.180193 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:43.180724 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:43.169472 4637 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:43.159302 4726 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260924 00:22:43.172770 4748 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260924 00:22:43.227766 1996 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:43.236442 4736 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260924 00:22:43.239020 4759 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260924 00:22:43.247044 2066 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260924 00:22:43.182770 2136 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:43.265026 2136 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:43.267699 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:43.268537 1996 raft_consensus.cc:3032] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:43.268901 1996 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:43.269763 1996 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:43.273540 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:43.274209 2063 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:43.284529 1996 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:43.292003 2063 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:43.299676 1996 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
W20260924 00:22:43.299691 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 130504us
I20260924 00:22:43.300863 2110 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:43.299017 2136 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.267725 4718 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260924 00:22:43.313512 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:43.321292 4700 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:43.331455 4726 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:43.332051 4726 raft_consensus.cc:686] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:43.332886 4726 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:43.333907 2649 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.342265 4768 raft_consensus.cc:979] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:43.342775 4699 raft_consensus.cc:1067] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:43.344899 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:43.353173 2036 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:43.399600 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 45258.3us
W20260924 00:22:43.400416 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 47363.7us
I20260924 00:22:43.405853 2136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:57748
I20260924 00:22:43.406476 2136 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:43.406816 2136 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:43.406972 4723 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.403105 4664 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:43.417373 2136 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:43.419687 2136 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:43.422099 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:43.433671 2144 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:43.436836 1893 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.457353 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:43.458227 1992 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:22:43.453845 2641 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.458567 1992 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:43.467907 1992 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:43.496397 1992 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:43.514061 1989 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
W20260924 00:22:43.548772 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 134415us
I20260924 00:22:43.551744 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:43.549628 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:43.562228 4779 raft_consensus.cc:979] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:43.562901 4779 raft_consensus.cc:1067] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:43.564543 4775 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:43.570787 4699 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:43.521720 1993 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.579286 1893 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.583509 2144 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:60184
I20260924 00:22:43.584101 2144 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:43.589847 4695 raft_consensus.cc:1050] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:22:43.593773 4695 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 4, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:43.587926 2144 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:43.602247 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:43.602885 2066 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:43.606549 2069 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260924 00:22:43.603143 1992 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:22:43.609282 2066 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:22:43.610234 2110 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:43.611028 4739 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:22:43.611488 4739 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:43.612277 4739 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:43.614629 2063 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.634447 1893 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 7, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:43.624372 2144 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:43.640899 4768 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:43.647882 2144 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:43.648932 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:43.650156 2069 raft_consensus.cc:3032] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:43.650434 2069 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:43.651051 2069 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:43.654659 2069 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:43.658252 2136 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260924 00:22:43.659207 2066 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:43.670598 2069 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:43.671828 2109 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:43.671075 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:43.677912 2641 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:43.675995 4723 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:43.669013 4699 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260924 00:22:43.670665 4712 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260924 00:22:43.696655 2641 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:43.715770 2646 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260924 00:22:43.698887 4723 raft_consensus.cc:686] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:43.718822 4723 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:43.721920 4699 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:43.731980 2069 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260924 00:22:43.759797 4606 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:43.766834 1893 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 7 to 8, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 8 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:43.770147 1891 catalog_manager.cc:5495] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 9 to 10, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 10 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:43.775436 2144 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260924 00:22:43.793898 2143 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:43.804414 4606 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260924 00:22:43.794701 4699 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260924 00:22:43.826745 2143 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:43.833827 2641 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:43.837469 2063 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:43.839401 4727 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:43.840528 1890 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 7 to 8, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 8 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:43.856837 2062 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:43.860232 4723 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:43.842707 4734 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260924 00:22:43.873839 4789 log.cc:924] Time spent T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260924 00:22:43.895838 2641 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.910326 1893 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 5 to 6, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 6 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:43.943902 2064 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:43.960397 4787 log.cc:924] Time spent T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260924 00:22:43.972600 2646 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:43.905845 4727 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:43.977532 4655 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:43.955487 1891 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 6 to 7, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.038918 1987 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260924 00:22:44.049713 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 57630.3us
W20260924 00:22:44.050100 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 58858.6us
I20260924 00:22:44.059895 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 708537d593c14817accda67f9fd081b4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:22:44.090656 4655 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.093247 2646 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.146665 2070 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.141727 2069 raft_consensus.cc:2932] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.111400 4616 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.165505 4748 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:44.204185 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet 708537d593c14817accda67f9fd081b4 with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55036
I20260924 00:22:44.210253 2069 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260924 00:22:44.212430 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 46002.4us
I20260924 00:22:44.217329 1987 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260924 00:22:44.265453 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 52961us
I20260924 00:22:44.270730 1992 raft_consensus.cc:2932] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
W20260924 00:22:44.283020 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 55223.5us
I20260924 00:22:44.293603 4739 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260924 00:22:44.323770 4795 log.cc:924] Time spent T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260924 00:22:44.325701 4802 tablet_replica.cc:330] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:22:44.340032 1893 catalog_manager.cc:5495] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.296887 4715 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260924 00:22:44.356383 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 54b1b238378448a5987c49aafc3b6313 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260924 00:22:44.367758 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 94532.7us
W20260924 00:22:44.441697 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 49705.5us
W20260924 00:22:44.611742 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 243094us
I20260924 00:22:44.639807 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:44.642364 2136 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.660140 2063 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:22:44.670137 2645 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:22:44.679109 2646 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.682767 1892 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 6 to 7, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:44.690387 4802 raft_consensus.cc:2229] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:44.706365 4802 raft_consensus.cc:2258] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Raft consensus is shut down!
W20260924 00:22:44.732645 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 55861.3us
I20260924 00:22:44.714810 4733 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
W20260924 00:22:44.856710 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 115327us
I20260924 00:22:44.859074 1992 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260924 00:22:44.867244 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 133757us
I20260924 00:22:44.837107 4642 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260924 00:22:44.919068 4802 ts_tablet_manager.cc:1905] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260924 00:22:44.945957 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 56372.5us
I20260924 00:22:44.952960 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:44.953887 2143 raft_consensus.cc:595] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:45.008571 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 44277.5us
W20260924 00:22:45.016635 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 49305.7us
W20260924 00:22:45.087196 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 125634us
W20260924 00:22:45.107512 4820 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20260924 00:22:45.140242 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 93582.7us
I20260924 00:22:45.159045 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ff940e15e1849efb194c908520c212f"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:45.159739 2136 raft_consensus.cc:595] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:45.146577 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 181229us
I20260924 00:22:45.176633 4723 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:45.192854 4830 raft_consensus.cc:979] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:45.193442 4830 raft_consensus.cc:1067] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:45.200050 1990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:45.200657 1990 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:45.200996 1990 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:45.206813 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:45.207486 1992 raft_consensus.cc:595] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:45.231693 1990 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.237280 1990 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:45.246762 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 197757us
I20260924 00:22:45.308297 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33238 (request call id 270) took 1081 ms. Trace:
I20260924 00:22:45.309173 1893 rpcz_store.cc:276] 0924 00:22:44.226757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:44.229913 (+ 3156us) service_pool.cc:227] Handling call
0924 00:22:45.308199 (+1078286us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:44.629422 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:44.704233 (+ 74811us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:44.707302 (+ 3069us) write_op.cc:435] Acquired schema lock
0924 00:22:44.709913 (+ 2611us) tablet.cc:660] Decoding operations
0924 00:22:44.738056 (+ 28143us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:44.741487 (+ 3431us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:44.743959 (+ 2472us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:44.790151 (+ 46192us) tablet.cc:699] Row locks acquired
0924 00:22:44.790173 (+ 22us) write_op.cc:260] PREPARE: finished
0924 00:22:44.798004 (+ 7831us) write_op.cc:270] Start()
0924 00:22:44.798142 (+ 138us) write_op.cc:276] Timestamp: P: 1790209364797948 usec, L: 0
0924 00:22:44.798165 (+ 23us) op_driver.cc:331] REPLICATION: starting
0924 00:22:44.798776 (+ 611us) log.cc:842] Serialized 591 byte log entry
0924 00:22:45.010700 (+211924us) op_driver.cc:444] REPLICATION: finished
0924 00:22:45.235739 (+225039us) write_op.cc:301] APPLY: starting
0924 00:22:45.235815 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:45.235908 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:45.236028 (+ 120us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:45.236210 (+ 182us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:45.236375 (+ 165us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:45.236415 (+ 40us) write_op.cc:312] APPLY: finished
0924 00:22:45.236724 (+ 309us) log.cc:842] Serialized 23 byte log entry
0924 00:22:45.236883 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:45.236952 (+ 69us) write_op.cc:454] Released schema lock
0924 00:22:45.236989 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:45.237294 (+ 305us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:45.237323 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":220564,"apply.run_cpu_time_us":1720,"apply.run_wall_time_us":1813,"num_ops":1,"prepare.queue_time_us":391,"prepare.run_cpu_time_us":15182,"prepare.run_wall_time_us":435946,"replication_time_us":212423,"thread_start_us":134496,"threads_started":1}]]}
I20260924 00:22:45.316917 4810 raft_consensus.cc:979] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:45.317435 4695 raft_consensus.cc:1067] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:45.321610 4802 ts_tablet_manager.cc:1918] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.6
I20260924 00:22:45.322187 4802 log.cc:1196] T 708537d593c14817accda67f9fd081b4 P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/708537d593c14817accda67f9fd081b4
I20260924 00:22:45.327111 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:45.327740 2143 raft_consensus.cc:3032] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:45.328025 2143 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:45.328900 2143 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.331337 2062 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:22:45.331815 2062 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:45.332096 2062 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:45.340935 2144 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:45.341884 2144 raft_consensus.cc:595] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:45.344573 4824 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:45.353709 4824 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.361905 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b1b238378448a5987c49aafc3b6313 with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57740
I20260924 00:22:45.362474 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:45.363175 2641 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:45.370792 4802 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55036 (request call id 593) took 1194 ms. Trace:
I20260924 00:22:45.371417 4802 rpcz_store.cc:276] 0924 00:22:44.195142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:44.203861 (+ 8719us) service_pool.cc:227] Handling call
0924 00:22:44.325310 (+121449us) ts_tablet_manager.cc:1145] Deleting tablet 708537d593c14817accda67f9fd081b4
0924 00:22:44.325410 (+ 100us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0924 00:22:45.166853 (+841443us) tablet_metadata.cc:641] Metadata flushed
0924 00:22:45.321339 (+154486us) tablet_metadata.cc:641] Metadata flushed
0924 00:22:45.364628 (+ 43289us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8069,"tablet-delete.queue_time_us":120424,"thread_start_us":70155,"threads_started":1}
I20260924 00:22:45.382563 2641 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:45.392426 2062 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:45.402740 2062 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:45.437718 4843 tablet_replica.cc:330] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
W20260924 00:22:45.439644 4840 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260924 00:22:45.454303 1990 raft_consensus.cc:1226] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:45.454659 2143 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:45.455230 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 57086us
I20260924 00:22:45.457275 4843 raft_consensus.cc:2229] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:45.463469 4824 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:45.464351 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:45.488591 2143 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:45.493234 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:45.512853 4830 raft_consensus.cc:979] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:45.523144 4824 raft_consensus.cc:1067] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:45.522557 4642 raft_consensus.cc:979] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:45.525627 4843 raft_consensus.cc:2258] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260924 00:22:45.527140 2138 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:45.527661 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:45.529459 4695 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:45.530305 4695 raft_consensus.cc:686] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:45.531734 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:45.531445 4695 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.532522 4726 raft_consensus.cc:1067] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:45.521185 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet 708537d593c14817accda67f9fd081b4 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:22:45.494076 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:45.536937 4843 ts_tablet_manager.cc:1905] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:22:45.537885 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:45.541174 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:45.543867 2109 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:45.553745 2109 raft_consensus.cc:2670] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260924 00:22:45.548183 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:45.562814 2066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:45.563458 2066 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:45.563843 2066 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:45.565361 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:45.566054 1992 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:45.566495 1992 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:45.566887 2062 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ff940e15e1849efb194c908520c212f"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:45.567391 2062 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:45.567816 2062 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:45.567339 1992 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:45.569038 1992 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:45.584195 2138 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:22:45.606819 2035 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:45.613090 1992 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:22:45.626823 2062 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.621513 2066 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:45.630976 2066 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:45.637346 2062 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:45.641501 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:45.642095 1988 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:45.643395 1992 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.641952 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:45.649732 2646 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:45.630641 4779 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:45.650331 4779 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:45.651681 4779 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:45.652520 1988 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:45.658914 2646 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:45.668177 2035 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:45.685395 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:45.686116 2138 raft_consensus.cc:3032] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:45.686414 2138 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:45.687093 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:45.687680 2140 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:45.687956 2140 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:45.692113 2138 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.683666 4748 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:45.701560 2138 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:45.710538 2036 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:45.705854 2140 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:45.722386 2138 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:45.709833 4748 raft_consensus.cc:686] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:45.752389 4843 ts_tablet_manager.cc:1918] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20260924 00:22:45.755110 4748 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:45.723367 4822 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:45.761323 4822 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:45.762498 4822 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:45.796366 2140 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:45.800499 1893 catalog_manager.cc:5495] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:45.755529 4843 log.cc:1196] T 54b1b238378448a5987c49aafc3b6313 P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/54b1b238378448a5987c49aafc3b6313
I20260924 00:22:45.835229 2140 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
W20260924 00:22:45.851971 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 143397us
I20260924 00:22:45.879086 2646 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260924 00:22:45.896245 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 54b1b238378448a5987c49aafc3b6313 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:22:45.897478 2139 raft_consensus.cc:1261] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260924 00:22:45.906342 4699 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:45.909991 1989 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:45.924726 4851 consensus_queue.cc:1035] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:46.003599 4851 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:46.018440 2646 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:46.020332 1891 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 8 to 9, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 9 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:46.029217 1891 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 6 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:46.031067 1891 catalog_manager.cc:5495] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 7 to 8, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 8 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:46.035925 2138 raft_consensus.cc:2932] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:46.042160 2649 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:46.044144 4616 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:46.052491 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:22:46.069743 1891 catalog_manager.cc:5495] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:46.093480 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47762
I20260924 00:22:46.116451 2138 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:22:46.117921 4851 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:46.123446 4863 tablet_replica.cc:330] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:22:46.124516 4863 raft_consensus.cc:2229] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:46.135099 4863 raft_consensus.cc:2258] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Raft consensus is shut down!
I20260924 00:22:46.142097 4863 ts_tablet_manager.cc:1905] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:22:46.167109 1992 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:46.168838 4799 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:46.175827 4863 ts_tablet_manager.cc:1918] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 9.7
I20260924 00:22:46.176538 4863 log.cc:1196] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/47c6ea1617904bf9bf2b2d904d0e2ae5
I20260924 00:22:46.178258 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet 47c6ea1617904bf9bf2b2d904d0e2ae5 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:22:46.209295 2138 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:46.211736 4748 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:46.255678 1992 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:46.261497 4780 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260924 00:22:46.278066 2649 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:46.285753 4682 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260924 00:22:46.310670 2143 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:46.314127 2138 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:46.316075 4822 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:46.316303 4748 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:46.809564 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:46.893193 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:46.894234 2143 raft_consensus.cc:595] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:46.906836 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:46.909579 1992 raft_consensus.cc:595] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:46.916498 2063 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:46.917094 2063 raft_consensus.cc:595] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:46.920751 2063 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:46.921444 2063 raft_consensus.cc:595] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:46.921754 4723 raft_consensus.cc:979] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:46.922660 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:46.923098 4642 raft_consensus.cc:1067] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:46.924631 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:46.926800 1992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:46.927501 1992 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:46.927974 1992 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:46.933452 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:46.941763 1992 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:46.946707 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:46.947319 2143 raft_consensus.cc:3032] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:46.947592 2143 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:46.948271 2143 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:46.949431 2143 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:46.960117 2143 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:46.964645 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:46.965263 2649 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:46.963222 1992 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:46.972527 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:46.973559 4699 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:46.974048 4699 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:46.974880 4699 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:46.977264 2649 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:47.023299 1893 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:47.217003 2062 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:47.226636 1992 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:47.227257 2138 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260924 00:22:47.230414 2035 consensus_peers.cc:487] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80ac2f44b186407ba12e93016e1ae270. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:47.235383 4748 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:47.237561 4879 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:47.248538 4685 raft_consensus.cc:979] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:47.249063 4685 raft_consensus.cc:1067] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:47.260385 2139 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:60184
I20260924 00:22:47.261118 2139 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:47.261556 2139 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:47.268719 2139 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.271497 2139 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:47.272135 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:47.272763 1992 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:47.280298 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:47.280912 2062 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:47.281239 2062 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:47.281953 2062 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.283186 2062 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:47.290660 2062 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:47.293040 2109 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:47.294204 4723 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:47.295024 4723 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:47.296057 4723 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.310532 1992 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:47.311513 4879 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:47.345960 4606 raft_consensus.cc:979] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:47.346446 4606 raft_consensus.cc:1067] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:47.347911 2143 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:47.349014 1893 catalog_manager.cc:5495] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 8 to 9, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 9 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:47.352051 2139 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:57748
I20260924 00:22:47.352626 2139 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:47.352972 2139 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:47.362223 2139 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.364112 2139 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:47.366003 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:47.382853 1992 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:47.383217 1992 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:47.383965 1992 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.375125 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:47.385645 2068 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:47.393682 2143 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:47.395819 1992 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:47.411401 2068 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:47.412668 2109 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:47.413623 4723 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:47.414146 4723 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:47.415045 4723 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.408906 1893 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 9 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:47.420653 1992 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:47.434931 1996 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Committing config change with OpId 9.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
W20260924 00:22:47.441154 2109 consensus_peers.cc:487] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 970b194b81a34f3db37adc13d58b0a01. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:47.488866 4879 raft_consensus.cc:979] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:47.489464 4879 raft_consensus.cc:1067] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:47.493528 1989 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:36934
I20260924 00:22:47.494097 1989 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:47.494481 1989 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:47.499102 1890 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 8 to 9, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 9 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:47.501734 2138 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:47.507085 4699 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:47.521054 1996 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:47.524082 4642 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:47.531929 1989 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.540995 1989 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:47.541733 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:47.541955 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:47.542495 2068 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:22:47.542821 2068 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:47.543571 2068 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.545044 2068 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:47.542835 2139 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:47.562598 2068 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
I20260924 00:22:47.563639 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:47.562642 4888 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:47.563654 1996 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:47.569887 2068 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:47.571231 2139 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
I20260924 00:22:47.572774 4616 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:22:47.575795 2646 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:47.593310 4616 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:47.594394 4616 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:47.591918 4723 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260924 00:22:47.646238 4699 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260924 00:22:47.679301 1893 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:47.682139 2139 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
W20260924 00:22:47.731829 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 91748us
I20260924 00:22:47.783177 4899 ts_tablet_manager.cc:927] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:47.783861 2068 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:47.840572 4899 tablet_copy_client.cc:316] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:22:47.843364 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 80ac2f44b186407ba12e93016e1ae270 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:22:47.843998 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 80ac2f44b186407ba12e93016e1ae270 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-80ac2f44b186407ba12e93016e1ae270
W20260924 00:22:47.860275 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 79704.6us
I20260924 00:22:47.867388 2081 tablet_copy_source_session.cc:205] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:47.872382 4899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ac2f44b186407ba12e93016e1ae270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:47.877045 2062 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:47.880493 4873 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260924 00:22:47.882210 2109 consensus_peers.cc:487] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed845c50caec4187b2daed75db23bfd8. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:47.875483 1890 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 9 to 10, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 10 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:47.914163 4899 tablet_copy_client.cc:791] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:47.914404 2066 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:47.915015 4899 tablet_copy_client.cc:655] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:47.917196 4723 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:47.921921 4899 tablet_copy_client.cc:531] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:47.941785 1996 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:47.960477 4723 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:47.975966 1996 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:47.978655 4899 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:47.988008 1996 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260924 00:22:47.993428 4900 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:48.001056 4873 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260924 00:22:48.015305 4900 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:48.024734 2068 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260924 00:22:48.027228 4907 ts_tablet_manager.cc:927] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:48.026120 2063 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:48.026873 4642 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:48.052937 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:48.081223 4907 tablet_copy_client.cc:316] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:22:48.118520 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet 970b194b81a34f3db37adc13d58b0a01 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:22:48.119120 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet 970b194b81a34f3db37adc13d58b0a01 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-970b194b81a34f3db37adc13d58b0a01
I20260924 00:22:48.132879 2155 tablet_copy_source_session.cc:205] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:48.137777 4907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970b194b81a34f3db37adc13d58b0a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:48.163067 2139 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260924 00:22:48.177433 1892 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:48.189528 1989 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:48.209527 2068 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260924 00:22:48.215508 4606 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:48.228834 4907 tablet_copy_client.cc:791] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:48.234936 4907 tablet_copy_client.cc:655] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:48.231514 2063 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:48.235551 4606 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:48.246459 4907 tablet_copy_client.cc:531] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:48.271108 4899 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260924 00:22:48.271263 2035 consensus_peers.cc:487] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4a9343576d1478caf59e9f5629ce211. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:48.272318 4899 tablet_bootstrap.cc:492] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:48.273053 4899 ts_tablet_manager.cc:1397] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.295s user 0.025s sys 0.022s
I20260924 00:22:48.279636 2140 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260924 00:22:48.282361 4748 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:48.291973 4899 raft_consensus.cc:348] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:48.292991 4899 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:22:48.294039 4899 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:48.297298 4899 ts_tablet_manager.cc:1428] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20260924 00:22:48.312172 1992 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260924 00:22:48.322927 4884 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:48.328301 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-80ac2f44b186407ba12e93016e1ae270 received from {username='slave'} at 127.0.0.1:43194
I20260924 00:22:48.328805 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-80ac2f44b186407ba12e93016e1ae270 on tablet 80ac2f44b186407ba12e93016e1ae270 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:48.379062 4899 ts_tablet_manager.cc:927] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:48.392592 4907 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
W20260924 00:22:48.432696 4919 log.cc:924] Time spent T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260924 00:22:48.438055 4884 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:48.445293 4899 tablet_copy_client.cc:316] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.3:36385
I20260924 00:22:48.447216 2155 tablet_copy_service.cc:140] P 8c129c01e1c047628f6a87b001aa7258: Received BeginTabletCopySession request for tablet ed845c50caec4187b2daed75db23bfd8 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:39164)
I20260924 00:22:48.447775 2155 tablet_copy_service.cc:161] P 8c129c01e1c047628f6a87b001aa7258: Beginning new tablet copy session on tablet ed845c50caec4187b2daed75db23bfd8 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:39164: session id = 9e79063532384ada8ea8f8205478adcf-ed845c50caec4187b2daed75db23bfd8
I20260924 00:22:48.456373 2137 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:48.474476 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85bede790e49439d97e8031810eee902"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:48.479971 1992 raft_consensus.cc:595] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:48.482151 2155 tablet_copy_source_session.cc:205] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:48.488992 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:48.490015 1992 raft_consensus.cc:595] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:48.494009 4899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed845c50caec4187b2daed75db23bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:48.507042 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:48.507681 2138 raft_consensus.cc:595] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:48.514369 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 71543.6us
I20260924 00:22:48.509601 1893 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 7, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 8 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:48.523380 2137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:48.528173 2137 raft_consensus.cc:595] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:48.532325 1992 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index -1 to 7, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:48.537259 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:48.538975 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:48.539696 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:48.558894 4907 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:22:48.571444 4907 tablet_bootstrap.cc:492] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:48.572175 4907 ts_tablet_manager.cc:1397] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.187s user 0.026s sys 0.008s
I20260924 00:22:48.598765 4907 raft_consensus.cc:348] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:48.604991 4907 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:22:48.606920 4899 tablet_copy_client.cc:791] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:48.607525 4899 tablet_copy_client.cc:655] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:48.606240 4907 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:48.611130 4907 ts_tablet_manager.cc:1428] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20260924 00:22:48.611546 4899 tablet_copy_client.cc:531] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:48.613368 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-970b194b81a34f3db37adc13d58b0a01 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:22:48.613865 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-970b194b81a34f3db37adc13d58b0a01 on tablet 970b194b81a34f3db37adc13d58b0a01 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:48.632275 4899 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:48.651346 2649 raft_consensus.cc:1203] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Deduplicated request from leader. Original: 9.5->[9.6-9.6] Dedup: 9.6->[]
I20260924 00:22:48.707192 4699 raft_consensus.cc:979] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:48.707669 4699 raft_consensus.cc:1067] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:48.709209 2140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:57748
I20260924 00:22:48.709770 2140 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:48.710109 2140 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:48.717391 4925 raft_consensus.cc:979] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:48.718027 4873 raft_consensus.cc:1067] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:48.723428 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:48.723943 2068 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:48.724268 4873 raft_consensus.cc:979] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:48.724313 2068 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:48.724797 4925 raft_consensus.cc:1067] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:48.726691 1992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:48.727198 1992 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:48.727507 1992 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:48.728416 2140 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:48.738253 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:48.737378 2140 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:48.739017 2649 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:48.739929 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:48.740619 1989 raft_consensus.cc:3032] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:48.740952 1989 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:48.741706 1989 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:48.742854 1989 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:48.746055 4907 ts_tablet_manager.cc:927] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:48.746611 2649 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:48.739709 4899 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:22:48.747768 2109 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:48.748615 4873 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:48.749284 4873 raft_consensus.cc:686] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:48.750545 4873 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:48.738876 2068 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:48.755539 4899 tablet_bootstrap.cc:492] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:48.756572 4899 ts_tablet_manager.cc:1397] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.124s user 0.030s sys 0.008s
I20260924 00:22:48.759418 4899 raft_consensus.cc:348] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:48.760301 4899 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:22:48.752789 1992 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:48.760967 4899 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:48.762948 2068 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:48.765184 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:48.776324 2138 raft_consensus.cc:3032] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:48.776643 2138 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:48.777282 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:48.777804 2066 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:48.777415 2138 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:48.778733 2138 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:48.766438 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:48.781424 2649 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:48.783809 1890 catalog_manager.cc:5495] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 4 to 5, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:48.767203 1989 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:48.786911 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:48.787545 2140 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Stepping down as leader of term 10
I20260924 00:22:48.787849 2140 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:48.791671 2649 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:48.772398 4907 tablet_copy_client.cc:316] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:22:48.793320 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:48.790395 2140 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:48.794646 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet e4a9343576d1478caf59e9f5629ce211 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:22:48.794942 4884 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:48.795252 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet e4a9343576d1478caf59e9f5629ce211 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-e4a9343576d1478caf59e9f5629ce211
I20260924 00:22:48.795675 4884 raft_consensus.cc:686] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:48.796578 4884 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:48.800159 4899 ts_tablet_manager.cc:1428] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260924 00:22:48.800490 2140 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:48.802019 2155 tablet_copy_service.cc:342] P 8c129c01e1c047628f6a87b001aa7258: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-ed845c50caec4187b2daed75db23bfd8 received from {username='slave'} at 127.0.0.1:39164
I20260924 00:22:48.802238 2138 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:48.802430 2155 tablet_copy_service.cc:433] P 8c129c01e1c047628f6a87b001aa7258: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-ed845c50caec4187b2daed75db23bfd8 on tablet ed845c50caec4187b2daed75db23bfd8 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:48.795121 2066 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
I20260924 00:22:48.809490 1992 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:48.814081 1960 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:48.817420 4895 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:22:48.820398 2140 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
I20260924 00:22:48.818046 4895 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:48.848621 2081 tablet_copy_source_session.cc:205] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:48.855018 4895 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:48.869735 4907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a9343576d1478caf59e9f5629ce211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:48.879618 4699 raft_consensus.cc:979] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:48.880167 4699 raft_consensus.cc:1067] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:48.882684 2066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "85bede790e49439d97e8031810eee902"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:22:48.886128 2066 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:48.886499 2066 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:48.901422 1890 catalog_manager.cc:5495] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 10 to 11, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 11 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:48.910980 2066 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:48.916210 2066 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:48.913534 1893 catalog_manager.cc:5495] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:48.919353 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:48.919957 2140 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:48.927608 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:48.928259 1992 raft_consensus.cc:3032] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:48.928550 1992 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:48.929283 1992 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:48.930480 1992 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:48.941754 2140 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:48.943004 1992 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:48.945580 2649 raft_consensus.cc:1203] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Deduplicated request from leader. Original: 8.5->[8.6-8.6] Dedup: 8.6->[]
I20260924 00:22:48.949858 2036 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:48.950778 4926 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:48.951275 4926 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:48.952066 4926 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:48.995482 4907 tablet_copy_client.cc:791] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:22:48.996109 4907 tablet_copy_client.cc:655] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:22:49.002679 4907 tablet_copy_client.cc:531] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:22:49.005993 2641 raft_consensus.cc:1203] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 LEARNER]: Deduplicated request from leader. Original: 7.5->[7.6-7.6] Dedup: 7.6->[]
I20260924 00:22:49.008387 1893 catalog_manager.cc:5495] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 6 to 7, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:49.022581 4907 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:22:49.080165 4907 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:22:49.081184 4907 tablet_bootstrap.cc:492] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:22:49.081921 4907 ts_tablet_manager.cc:1397] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 0.060s user 0.028s sys 0.013s
I20260924 00:22:49.084937 4907 raft_consensus.cc:348] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:49.085841 4907 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:22:49.086546 4907 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:49.089345 4907 ts_tablet_manager.cc:1428] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 00:22:49.107322 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-e4a9343576d1478caf59e9f5629ce211 received from {username='slave'} at 127.0.0.1:43194
I20260924 00:22:49.107752 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-e4a9343576d1478caf59e9f5629ce211 on tablet e4a9343576d1478caf59e9f5629ce211 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:49.224658 4884 raft_consensus.cc:1050] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:22:49.227717 4884 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 4, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:49.259123 2641 raft_consensus.cc:1203] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Deduplicated request from leader. Original: 8.6->[8.7-8.7] Dedup: 8.7->[]
I20260924 00:22:49.272598 2641 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260924 00:22:49.274035 1989 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260924 00:22:49.274205 4926 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:49.277163 4884 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:49.300426 2139 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260924 00:22:49.304260 4884 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:49.309914 2139 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260924 00:22:49.311856 4699 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:49.314245 4748 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:49.320876 2641 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:49.321949 1989 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:49.323308 4873 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:49.332573 2641 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:49.341073 2138 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:49.342748 4873 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:49.346733 2649 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:49.351594 1990 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:49.352931 4921 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:49.374740 4884 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:49.415295 2138 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:49.412505 1890 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 9 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260924 00:22:49.512082 4939 log.cc:924] Time spent T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260924 00:22:49.521698 4925 raft_consensus.cc:1050] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:22:49.528832 2066 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260924 00:22:49.524875 4925 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 4, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:49.561069 4723 raft_consensus.cc:1050] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:22:49.568102 1992 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:49.577328 4699 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:49.591281 2066 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:49.563580 4723 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 4, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:49.633446 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 79823.4us
W20260924 00:22:49.650974 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 75249.2us
I20260924 00:22:49.659518 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260924 00:22:49.699503 4936 log.cc:924] Time spent T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
W20260924 00:22:49.751809 4940 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20260924 00:22:49.767573 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 105480us
W20260924 00:22:49.798391 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 71170us
I20260924 00:22:49.833290 2641 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260924 00:22:49.848255 2137 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:49.848570 2649 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260924 00:22:49.852677 4966 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:49.854785 4956 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260924 00:22:49.892474 4974 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:49.896847 2641 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 LEARNER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260924 00:22:49.898815 2062 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260924 00:22:49.904660 4723 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:49.923604 1990 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260924 00:22:49.956924 2066 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260924 00:22:49.961022 4969 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:49.969383 1989 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260924 00:22:49.975037 2062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:49.975869 2062 raft_consensus.cc:595] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:49.978513 4969 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:49.991559 1987 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260924 00:22:50.006073 4962 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:50.011574 4723 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 00:22:50.024456 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 53635.1us
I20260924 00:22:50.032616 4974 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260924 00:22:50.074950 4748 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:50.094966 4974 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.097802 4884 raft_consensus.cc:1050] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: attempting to promote NON_VOTER 9e79063532384ada8ea8f8205478adcf to VOTER
I20260924 00:22:50.100301 4884 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:50.106424 4962 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.109854 2641 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.118338 4723 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.122200 2069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:50.122793 2069 raft_consensus.cc:595] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:50.124094 4988 log.cc:924] Time spent T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.057s user 0.003s sys 0.000s
I20260924 00:22:50.130085 2139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:50.131683 2139 raft_consensus.cc:595] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:50.130045 2646 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.135953 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 80ac2f44b186407ba12e93016e1ae270 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:22:50.136197 1993 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:50.136812 1993 raft_consensus.cc:595] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:50.140272 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260924 00:22:50.141903 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:50.142495 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:50.150177 4921 raft_consensus.cc:979] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:50.150660 4921 raft_consensus.cc:1067] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:50.145769 1992 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.163308 1993 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:36934
I20260924 00:22:50.163878 1993 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:50.164238 1993 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:50.170537 1990 raft_consensus.cc:2932] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Committing config change with OpId 9.8: config changed from index 7 to 8, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
W20260924 00:22:50.171629 4993 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260924 00:22:50.175961 1988 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:50.177428 4974 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:50.201073 2646 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 LEARNER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:50.205960 2140 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:50.210340 4748 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:50.211870 1892 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 7 to 8, VOTER 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) evicted. New cstate: current_term: 9 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.241236 4957 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:50.233676 1993 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:50.297425 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 62675.5us
W20260924 00:22:50.361162 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 69541.8us
I20260924 00:22:50.372272 2070 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.380098 1993 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:50.383881 1891 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:50.404521 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:50.405277 2062 raft_consensus.cc:3032] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:50.405578 2062 raft_consensus.cc:729] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:50.406459 2062 consensus_queue.cc:260] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.407575 2062 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:22:50.418535 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 168144us
I20260924 00:22:50.452436 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:50.453068 2137 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:50.456045 4973 raft_consensus.cc:979] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:50.456620 4973 raft_consensus.cc:1067] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:50.463737 2137 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:50.467391 2641 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.490478 2062 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:50.493238 4959 raft_consensus.cc:979] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:50.499362 4959 raft_consensus.cc:1067] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:50.503861 2145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:60184
I20260924 00:22:50.503243 1988 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.504443 2145 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:50.505324 2145 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:50.514091 1993 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:50.514968 1993 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:50.515484 1993 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:50.514149 1960 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:50.521765 2145 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.531149 4960 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:50.538602 4960 raft_consensus.cc:686] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:50.539497 4960 consensus_queue.cc:237] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.546892 2145 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:50.549124 2646 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.548308 2062 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.549834 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:50.554983 1987 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:50.544759 4981 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.561537 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:50.562222 2071 raft_consensus.cc:3032] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:50.562496 2071 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:50.563572 2071 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.565318 2071 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:50.568661 1987 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:50.568707 1993 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.571535 1993 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:50.571970 2110 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:50.575711 4978 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:50.576006 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:50.576305 4978 raft_consensus.cc:686] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:50.576639 2068 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:50.576509 2071 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:50.578862 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:50.579556 2138 raft_consensus.cc:3032] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:50.580955 2138 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:50.581898 2138 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.583184 2138 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:50.577332 4978 consensus_queue.cc:237] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.588197 2140 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.590008 2068 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:50.590871 2138 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:50.593288 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:50.594130 4606 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:50.594637 4606 raft_consensus.cc:686] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:50.595492 4606 consensus_queue.cc:237] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.626817 2062 raft_consensus.cc:1226] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:50.626507 1987 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.641068 4960 raft_consensus.cc:979] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:50.641594 4960 raft_consensus.cc:1067] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:50.643560 2140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:57748
I20260924 00:22:50.644208 2140 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:50.644583 2140 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:50.706974 2140 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.740093 2140 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:50.741008 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:50.741631 2068 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:50.742097 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:50.742652 1987 raft_consensus.cc:3032] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:50.742949 1987 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:50.743701 1987 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.744874 1987 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:50.749330 2068 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:50.750746 2109 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:50.751626 1987 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:50.752357 4978 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:50.753489 4978 raft_consensus.cc:686] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:50.754343 4978 consensus_queue.cc:237] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:50.787972 2132 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ac2f44b186407ba12e93016e1ae270 with delete_type TABLET_DATA_TOMBSTONED (TS 8c129c01e1c047628f6a87b001aa7258 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57740
I20260924 00:22:50.822090 5010 tablet_replica.cc:330] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: stopping tablet replica
I20260924 00:22:50.830145 5010 raft_consensus.cc:2229] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:50.831234 5010 raft_consensus.cc:2258] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Raft consensus is shut down!
I20260924 00:22:50.834326 5010 ts_tablet_manager.cc:1905] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:22:50.842451 1890 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 6 to 7, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.866041 1890 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index 7 to 8, 9e79063532384ada8ea8f8205478adcf (127.0.203.4) changed from NON_VOTER to VOTER. New cstate: current_term: 8 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:50.868711 5010 ts_tablet_manager.cc:1918] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 9.7
I20260924 00:22:50.878285 5010 log.cc:1196] T 80ac2f44b186407ba12e93016e1ae270 P 8c129c01e1c047628f6a87b001aa7258: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-2-root/wals/80ac2f44b186407ba12e93016e1ae270
I20260924 00:22:50.880232 1879 catalog_manager.cc:4841] TS 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385): tablet 80ac2f44b186407ba12e93016e1ae270 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:22:50.893463 1893 catalog_manager.cc:5495] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 5, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 5 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:50.895352 1893 catalog_manager.cc:5495] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 3 to 4, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 4 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:50.928130 2066 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:50.935164 1891 catalog_manager.cc:5495] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 7, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:50.942621 1891 catalog_manager.cc:5495] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 7, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:50.950340 2646 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260924 00:22:50.950410 2138 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260924 00:22:50.952401 4981 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:22:50.955060 2066 raft_consensus.cc:1261] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:50.956979 4981 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:50.966205 4606 consensus_queue.cc:1035] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:50.966360 4926 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Committing config change with OpId 8.9: config changed from index 8 to 9, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.967953 2649 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 8 to 9, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.986229 2062 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:50.989389 2138 raft_consensus.cc:2932] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 8 to 9, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:50.997200 4967 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:51.098670 2138 raft_consensus.cc:1261] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260924 00:22:51.098775 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e4a9343576d1478caf59e9f5629ce211 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:22:51.103782 2145 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:51.104048 2138 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:51.104681 1993 raft_consensus.cc:1261] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:51.106073 1890 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf reported cstate change: config changed from index 8 to 9, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 8 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
W20260924 00:22:51.122690 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 107044us
I20260924 00:22:51.126335 2646 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:51.143754 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a9343576d1478caf59e9f5629ce211 with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55036
I20260924 00:22:51.146847 5014 consensus_queue.cc:1035] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260924 00:22:51.152622 4962 consensus_queue.cc:1035] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:51.154402 2068 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:51.160326 2068 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:51.160573 2646 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:51.162209 1993 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:51.165212 5014 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260924 00:22:51.168197 2138 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:51.199375 5013 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260924 00:22:51.208513 5018 tablet_replica.cc:330] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:22:51.209787 5018 raft_consensus.cc:2229] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:51.208611 5013 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260924 00:22:51.221928 4967 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:51.233899 5015 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:51.234381 4978 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:51.283339 1993 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260924 00:22:51.277964 5018 raft_consensus.cc:2258] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Raft consensus is shut down!
I20260924 00:22:51.326454 2066 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:51.367383 5025 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Committing config change with OpId 8.8: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:51.369818 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:51.374938 5018 ts_tablet_manager.cc:1905] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:22:51.306591 5027 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260924 00:22:51.324301 5026 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
W20260924 00:22:51.562084 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 165928us
W20260924 00:22:51.585781 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 112072us
W20260924 00:22:51.594444 5003 log.cc:924] Time spent T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260924 00:22:51.627084 1988 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
W20260924 00:22:51.717782 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 311011us
W20260924 00:22:51.694906 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 85586.2us
W20260924 00:22:51.789793 5028 log.cc:924] Time spent T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.281s user 0.001s sys 0.000s
W20260924 00:22:51.803110 5029 log.cc:924] Time spent T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260924 00:22:51.823853 1988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:51.824535 1988 raft_consensus.cc:595] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:51.855706 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:51.856339 2140 raft_consensus.cc:595] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:51.860194 2646 raft_consensus.cc:2932] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:51.866035 2641 raft_consensus.cc:2932] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:51.891322 5018 ts_tablet_manager.cc:1918] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 8.8
I20260924 00:22:51.891870 5018 log.cc:1196] T e4a9343576d1478caf59e9f5629ce211 P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/e4a9343576d1478caf59e9f5629ce211
I20260924 00:22:51.896456 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:51.897094 1990 raft_consensus.cc:595] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:51.907027 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 71660.4us
I20260924 00:22:51.914608 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:51.916206 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 970b194b81a34f3db37adc13d58b0a01 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:22:51.922004 2140 raft_consensus.cc:595] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:51.925563 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
W20260924 00:22:51.925783 5025 consensus_meta.cc:287] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.556s user 0.011s sys 0.000s
I20260924 00:22:51.926913 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ed845c50caec4187b2daed75db23bfd8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260924 00:22:51.927479 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:51.928153 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:51.950037 1890 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: config changed from index 7 to 8, VOTER 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) evicted. New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:51.966679 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet e4a9343576d1478caf59e9f5629ce211 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:22:52.074807 2065 raft_consensus.cc:1261] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260924 00:22:52.084683 5053 raft_consensus.cc:979] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:52.085417 5069 raft_consensus.cc:1067] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Signalling peer 8c129c01e1c047628f6a87b001aa7258 to start an election
I20260924 00:22:52.091948 5046 raft_consensus.cc:979] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:52.092522 5046 raft_consensus.cc:1067] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Signalling peer 3eb374c9744b4c7e92dc749aece177dd to start an election
I20260924 00:22:52.096969 1988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:55052
I20260924 00:22:52.097609 1988 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:52.098014 1988 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:52.130719 2140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:57748
I20260924 00:22:52.131732 2140 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:52.132213 2140 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:52.141156 1988 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:52.144881 1988 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:52.165102 4967 raft_consensus.cc:979] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:52.166159 5054 raft_consensus.cc:1067] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:52.170428 5042 raft_consensus.cc:979] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:52.170902 5042 raft_consensus.cc:1067] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:52.172217 2071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47770
I20260924 00:22:52.172735 2071 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:52.173059 2071 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:52.173890 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:52.174423 2062 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:52.180194 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:52.180753 2137 raft_consensus.cc:3032] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:52.181044 2137 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:52.184237 2063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:22:52.184749 2063 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:22:52.185077 2063 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:52.185431 2140 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:52.186726 2071 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:52.181758 2137 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:52.193339 2062 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:52.145228 5025 consensus_queue.cc:1035] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260924 00:22:52.206564 2140 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:52.206770 2071 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:52.211460 2137 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:52.211820 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:52.212605 2140 raft_consensus.cc:3032] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:52.213083 2140 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:52.213837 2140 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:52.215492 2140 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:52.216719 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:52.217321 2641 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:52.212610 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:52.220461 2649 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:52.225639 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:52.237326 5052 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:52.237891 5052 raft_consensus.cc:686] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:52.238740 5052 consensus_queue.cc:237] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:52.245294 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:52.245957 1988 raft_consensus.cc:3032] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:52.246246 1988 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:52.248245 2140 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:52.252785 2641 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:52.257117 2137 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:52.249632 2649 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:22:52.260144 1988 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:52.261246 1890 catalog_manager.cc:5495] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: config changed from index 7 to 8, VOTER 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) evicted. New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:52.261584 2063 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:52.266032 2109 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:52.266335 1988 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:52.267486 2063 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:52.269986 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:52.272768 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:52.271699 4958 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:52.273432 2137 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:52.258039 1981 tablet_service.cc:1514] Processing DeleteTablet for tablet 970b194b81a34f3db37adc13d58b0a01 with delete_type TABLET_DATA_TOMBSTONED (TS 3eb374c9744b4c7e92dc749aece177dd not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55036
I20260924 00:22:52.275266 5018 tablet_replica.cc:330] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: stopping tablet replica
I20260924 00:22:52.275857 2035 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:52.276307 5018 raft_consensus.cc:2229] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:52.277994 5018 raft_consensus.cc:2258] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Raft consensus is shut down!
I20260924 00:22:52.279115 4959 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:52.279590 4959 raft_consensus.cc:686] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:52.280730 5018 ts_tablet_manager.cc:1905] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:22:52.286518 4958 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:52.287608 4958 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:52.298170 1987 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:22:52.299436 1987 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:52.280455 4959 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:52.303792 1987 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:52.303776 1988 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:52.322000 5018 ts_tablet_manager.cc:1918] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.7
I20260924 00:22:52.322777 2137 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:22:52.322289 1987 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:52.328794 2035 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:52.322536 5018 log.cc:1196] T 970b194b81a34f3db37adc13d58b0a01 P 3eb374c9744b4c7e92dc749aece177dd: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-0-root/wals/970b194b81a34f3db37adc13d58b0a01
I20260924 00:22:52.340554 1987 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:22:52.423769 1880 catalog_manager.cc:4841] TS 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): tablet 970b194b81a34f3db37adc13d58b0a01 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
W20260924 00:22:52.425243 5080 log.cc:924] Time spent T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.093s user 0.004s sys 0.013s
I20260924 00:22:52.428313 1890 catalog_manager.cc:5495] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:22:52.429471 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 87026.3us
I20260924 00:22:52.344770 5062 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:22:52.441875 1891 catalog_manager.cc:5495] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 7, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:52.437783 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet ed845c50caec4187b2daed75db23bfd8 with delete_type TABLET_DATA_TOMBSTONED (TS 948356bf66a941ef8249ecbd0ec94f6d not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47762
I20260924 00:22:52.454109 5062 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:52.455180 5062 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:52.463107 5055 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:52.463716 5055 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } attrs { replace: true } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:52.466706 5055 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:52.469937 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:52.472360 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:52.479889 2035 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:52.485345 5055 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:52.499395 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:52.521088 5084 tablet_replica.cc:330] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: stopping tablet replica
I20260924 00:22:52.528623 5084 raft_consensus.cc:2229] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Raft consensus shutting down.
I20260924 00:22:52.530191 5084 raft_consensus.cc:2258] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Raft consensus is shut down!
I20260924 00:22:52.533476 1890 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 9 to 10, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 10 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:52.538055 1890 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 7 to 8, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 8 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:52.541501 5084 ts_tablet_manager.cc:1905] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260924 00:22:52.576692 5084 ts_tablet_manager.cc:1918] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 8.7
I20260924 00:22:52.577261 5084 log.cc:1196] T ed845c50caec4187b2daed75db23bfd8 P 948356bf66a941ef8249ecbd0ec94f6d: Deleting WAL directory at /tmp/dist-test-taskTMSrdH/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790209224507772-812-0/minicluster-data/ts-1-root/wals/ed845c50caec4187b2daed75db23bfd8
I20260924 00:22:52.579227 1879 catalog_manager.cc:4841] TS 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): tablet ed845c50caec4187b2daed75db23bfd8 (table test-workload [id=f6f2ae7fecfa44b88e1797df02c71f60]) successfully deleted
I20260924 00:22:52.654824 2067 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:52.656245 5052 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:52.696087 2136 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:52.697538 4606 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:52.701555 2145 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260924 00:22:52.702842 5062 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:22:52.713908 2649 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260924 00:22:52.716500 5061 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:52.755319 1995 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:52.756865 4962 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:52.822461 2649 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:52.834426 4962 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:53.948870 5094 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:53.949801 5094 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:53.953099 5095 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:53.953796 5095 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:53.956511 5095 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:53.958117 5094 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:53.960181 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:53.961187 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:53.964109 5094 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:53.966380 5094 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:53.970383 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:53.973228 5094 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.002822 5094 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.003717 5094 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.006253 5095 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.007283 5095 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.046944 5094 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.049989 5094 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.067155 5094 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.078073 5097 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.078877 5097 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.079748 5094 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.080513 5094 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.081594 5097 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.081207 5094 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.084470 5096 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.087486 5096 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.090487 5096 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.091656 5096 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.094465 5096 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.097415 5052 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.098177 5096 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.098249 5052 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.099078 5096 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.091676 5097 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:54.102775 5096 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.110972 5096 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.108808 5097 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:54.108147 5052 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.114440 5097 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:54.115242 5094 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.122212 5094 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:54.122664 5094 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.125000 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2394) took 1171 ms. Trace:
I20260924 00:22:54.125478 1987 rpcz_store.cc:276] 0924 00:22:52.953713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.124130 (+1170417us) service_pool.cc:227] Handling call
0924 00:22:54.124450 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:54.124632 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:54.124657 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:54.124702 (+ 45us) raft_consensus.cc:1631] Marking committed up to 7
0924 00:22:54.124747 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:54.124813 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:54.124937 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:54.126581 5098 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.127120 5098 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.132985 5098 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.134124 5094 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.135268 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:54.136008 1987 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:54.136665 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:54.136859 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1741) took 1185 ms. Trace:
I20260924 00:22:54.137274 2067 rpcz_store.cc:276] 0924 00:22:52.951163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.136063 (+1184900us) service_pool.cc:227] Handling call
0924 00:22:54.136378 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:54.136551 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:54.136576 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:54.136612 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:22:54.136651 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:54.136707 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:54.136817 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:54.138149 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:54.138630 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:54.139264 1990 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:54.141394 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.142060 2066 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:54.143079 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.143671 2066 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:54.144510 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.144718 2036 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:54.145818 5094 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:54.145741 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.146287 5094 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.146690 5094 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:54.147044 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.147697 2065 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:54.150170 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.155537 5094 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.156749 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1744) took 1021 ms. Trace:
I20260924 00:22:54.156857 5099 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.157428 2071 rpcz_store.cc:276] 0924 00:22:53.134972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.155650 (+1020678us) service_pool.cc:227] Handling call
0924 00:22:54.156032 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:54.156312 (+ 280us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:54.156353 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:54.156408 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:54.156473 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:54.156545 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:54.156673 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:54.158856 5100 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:54.159521 5100 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.160118 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:54.162895 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:54.163985 5052 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:54.164198 5094 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.164397 5052 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.165001 5052 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:54.164237 5100 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.158365 5099 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.170265 5099 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.184022 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:54.229894 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 45974.5us
I20260924 00:22:54.191354 5097 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:54.185654 5098 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.233723 5099 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:54.241426 5097 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.251926 5104 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.252543 5104 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.254541 5104 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.244023 5099 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.259552 2036 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:54.250900 5097 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:54.244169 5098 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.262246 5104 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:54.255378 5101 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.259323 5099 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:54.258764 5096 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.263084 5094 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.282487 5105 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.282440 5094 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.283090 5105 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.284631 5094 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.285663 5096 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.287659 5096 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.291150 5098 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.291721 5098 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.292176 5098 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.293416 5094 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.281380 5104 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.293980 5094 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.295150 5104 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.284982 5105 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.383926 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.385766 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.386656 2067 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:54.422902 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:54.423835 2067 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:54.351557 5094 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.297847 5101 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.519941 5101 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.520604 5101 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.419555 5105 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:54.505391 5108 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.510139 5098 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.531206 5098 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:54.530980 5099 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.530824 5052 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.531680 5098 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.533020 5099 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.533492 5105 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.536247 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1742) took 1423 ms. Trace:
I20260924 00:22:54.536489 5105 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.512331 5094 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:54.536652 2063 rpcz_store.cc:276] 0924 00:22:53.113525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.535356 (+1421831us) service_pool.cc:227] Handling call
0924 00:22:54.535678 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:54.535867 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:54.535895 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:54.535934 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:22:54.535975 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:54.536027 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:54.536154 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:54.537391 5094 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.538074 5094 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:54.537313 5108 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.539954 5108 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.540910 5052 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.515038 5097 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:54.553468 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 293950us
I20260924 00:22:54.550745 5105 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.566026 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1746) took 1409 ms. Trace:
I20260924 00:22:54.566447 2068 rpcz_store.cc:276] 0924 00:22:53.156833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.565308 (+1408475us) service_pool.cc:227] Handling call
0924 00:22:54.565621 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:54.565747 (+ 126us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:54.565762 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:54.565786 (+ 24us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:54.565810 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:54.565901 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:54.565978 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:54.525077 5101 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.572872 5111 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.573186 5106 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:54.573467 5111 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.573715 5106 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.574824 5112 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.575595 5112 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.577086 5095 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.580636 5101 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.576210 5111 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.567879 5105 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.571194 5098 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.586836 5102 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:54.588171 5097 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.820523 5111 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:54.811400 5102 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:54.848330 5102 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:54.851922 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1749) took 1498 ms. Trace:
I20260924 00:22:54.852547 2069 rpcz_store.cc:276] 0924 00:22:53.353274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.850761 (+1497487us) service_pool.cc:227] Handling call
0924 00:22:54.851245 (+ 484us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:54.851477 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:54.851515 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:54.851570 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:54.851625 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:54.851693 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:54.851843 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:54.856917 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1750) took 1479 ms. Trace:
I20260924 00:22:54.857491 2064 rpcz_store.cc:276] 0924 00:22:53.377752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.855611 (+1477859us) service_pool.cc:227] Handling call
0924 00:22:54.856083 (+ 472us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:54.856331 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:54.856369 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:54.856423 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:22:54.856474 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:54.856555 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:54.856695 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:54.861423 5112 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.862150 5112 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.862740 5112 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.864023 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:54.865444 5106 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:54.866129 5106 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.866667 5106 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.875281 1991 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:54.879599 5105 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.837071 5111 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.887991 5111 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:54.888579 1961 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.025s sys 0.064s
I20260924 00:22:54.856519 5094 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:54.889124 1961 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.654s user 0.025s sys 0.064s
W20260924 00:22:54.904209 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 349283us
I20260924 00:22:54.890594 5106 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.890728 5112 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:54.892741 5094 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:54.914948 5052 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:54.942186 5113 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.011301 5121 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.011611 5052 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.011935 5113 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.012284 5121 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.013960 5052 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.018852 5123 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.013960 5113 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.019583 5123 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:54.910734 5097 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.022732 5128 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.018756 5127 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.019987 5113 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.015129 5112 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.014905 5121 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.022533 5097 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.023008 5123 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.014927 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1743) took 1900 ms. Trace:
I20260924 00:22:55.022533 5102 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.023345 5107 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.030058 5118 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.032508 5119 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.032725 5117 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.033216 5106 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:54.972097 5099 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.028836 5125 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.025032 5115 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.045523 2062 rpcz_store.cc:276] 0924 00:22:53.114271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:54.971287 (+1857016us) service_pool.cc:227] Handling call
0924 00:22:55.014455 (+ 43168us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:55.014628 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:55.014651 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:55.014687 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:55.014726 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:55.014776 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:55.014874 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:55.055778 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.059501 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.054978 5126 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.053766 5119 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.064159 5126 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.065800 5119 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.066197 5126 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.066334 5119 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.057171 5130 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.066854 5119 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.066956 5130 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.069013 5130 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.048326 5102 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.046531 5113 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.060954 2062 raft_consensus.cc:1226] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:55.060653 5115 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.070959 5102 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.059600 5117 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.072099 5115 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.031849 5128 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.072481 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.061933 5097 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.034830 5127 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.063000 2068 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:22:55.062556 5099 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.061985 2064 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:55.062786 5125 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.048184 5121 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.043455 5112 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.073436 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.074911 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.068863 5119 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.074954 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.103732 2062 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:55.053653 5118 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.077203 5113 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.077749 5126 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.104744 5126 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.105739 5118 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.105875 5122 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.106400 5131 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.106464 5122 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.107067 5131 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.108652 5122 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.051508 5123 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.109575 5132 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.109407 5123 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.110139 5132 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.111490 5123 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.083770 5133 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.109661 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 2240) took 1335 ms. Trace:
I20260924 00:22:55.107283 5126 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.118484 1994 rpcz_store.cc:276] 0924 00:22:53.773918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:55.108386 (+1334468us) service_pool.cc:227] Handling call
0924 00:22:55.108818 (+ 432us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:55.109041 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:55.109080 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:55.109152 (+ 72us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:22:55.109212 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:55.109281 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:55.109437 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:22:55.111832 5097 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.127666 5097 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.129914 5124 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.130595 5124 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.133363 5113 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:55.133958 5113 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.134728 5128 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.135417 5128 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.135954 5128 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.091722 5117 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.136935 5133 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.139178 5112 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.139468 5133 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.136314 5113 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.056078 5106 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.050997 5107 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.132992 5124 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.298419 5052 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.129936 5097 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.327513 5132 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.327898 5052 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.328934 5107 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.329118 5128 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.329694 5052 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.109040 5131 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.117723 5119 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.330924 5128 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.331485 5128 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.332274 5137 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.332851 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.140962 5099 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.333422 5128 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.333911 5128 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:22:55.220336 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 149568us
I20260924 00:22:55.334381 5128 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.336130 5111 consensus_queue.cc:1046] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:55.337189 5131 raft_consensus.cc:3032] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:55.337466 5128 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.337512 5131 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:55.146322 5125 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.338785 5128 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.332755 5137 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.340263 5128 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.340534 5137 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.340929 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.341463 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.342245 5128 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.338518 5131 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.342512 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.343170 5131 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:55.344118 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.343976 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.345024 2143 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:55.346436 2613 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:55.347867 5134 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:55.348248 5134 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.348579 2613 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:55.348587 5134 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:55.356545 5107 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.357034 5107 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.357982 5130 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:55.358361 5130 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.358701 5130 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:55.359057 5107 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.359562 5107 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.359995 5107 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.361439 5052 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.361948 5052 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.362351 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.363565 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.364691 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.365787 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.366849 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.351706 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.368008 2143 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:22:55.368647 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:55.368898 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:55.369483 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:55.369836 2065 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:55.370152 2071 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:55.369908 5130 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.370668 5107 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.370934 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.371179 5107 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.371531 2070 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:55.371939 5130 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.373629 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:55.374135 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.374219 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:55.375851 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.375756 5134 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.376467 2066 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:55.377470 2067 raft_consensus.cc:2379] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:55.111961 5121 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.377991 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.378616 2643 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:55.371723 5107 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.354444 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.379062 5121 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.355029 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.379604 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.112008 5122 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.363711 5052 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.380395 5122 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.378734 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:55.381325 2070 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:55.381488 5107 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.382328 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.383446 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.384634 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.352171 5099 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.385214 2063 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:55.385282 5099 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.125085 5118 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.387193 5099 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.387413 5118 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.389024 5118 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.350833 5131 consensus_queue.cc:1046] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:22:55.389508 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.390008 5131 raft_consensus.cc:3032] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:55.390403 5131 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:55.391165 5131 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.392361 5131 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:55.393767 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.394364 2649 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:55.391356 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.395467 2063 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:22:55.396672 1994 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:55.398172 4606 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.398720 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.398645 4606 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.400585 5052 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:55.401628 5137 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.401645 5052 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.403141 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.404196 5052 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.404703 1996 raft_consensus.cc:1226] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:55.405568 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.406143 1992 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:55.406314 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.404211 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.407226 1991 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 2.
I20260924 00:22:55.400499 4606 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.408056 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.408854 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.409446 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.409476 1987 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:22:55.410058 1989 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 9.
I20260924 00:22:55.410693 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.400516 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:55.402135 5137 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.412388 5130 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:55.412753 5130 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.412941 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.413125 5130 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:55.144567 5127 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.420516 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2393) took 2467 ms. Trace:
I20260924 00:22:55.413448 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.439688 1995 rpcz_store.cc:276] 0924 00:22:52.952919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:55.419501 (+2466582us) service_pool.cc:227] Handling call
0924 00:22:55.419901 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:22:55.420117 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:22:55.420153 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:22:55.420203 (+ 50us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:22:55.420259 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:22:55.420333 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:22:55.420450 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260924 00:22:55.442375 1925 debug-util.cc:398] Leaking SignalData structure 0x7b0800039260 after lost signal to thread 815
W20260924 00:22:55.443585 1925 debug-util.cc:398] Leaking SignalData structure 0x7b0800316620 after lost signal to thread 1928
W20260924 00:22:55.445079 1925 debug-util.cc:398] Leaking SignalData structure 0x7b08000331e0 after lost signal to thread 2020
I20260924 00:22:53.279235 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260924 00:22:55.446614 1925 debug-util.cc:398] Leaking SignalData structure 0x7b080008eb20 after lost signal to thread 2095
W20260924 00:22:55.452086 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.210s user 0.003s sys 0.016s
W20260924 00:22:55.452673 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.211s user 0.005s sys 0.018s
I20260924 00:22:55.416334 5137 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.455878 5099 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.428448 2065 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:55.456646 5099 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.460559 1992 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:55.460664 5099 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.457607 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.461443 1993 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:55.461812 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.462379 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.462913 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.463133 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.463387 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.463937 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.464412 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.466174 5102 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.466675 5102 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.467253 5140 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.467738 5140 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.468755 2649 raft_consensus.cc:1226] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:55.469678 5140 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.470148 5140 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.470525 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.470639 5140 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.427253 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:55.472265 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 45099.7us
I20260924 00:22:55.472507 5140 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.441607 5119 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.473297 5099 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:55.473703 5099 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.474068 5099 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:55.474274 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.474608 5119 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.473297 5102 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.464490 1992 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:22:55.475478 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.476133 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.430888 2070 raft_consensus.cc:2454] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:22:55.476589 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.476754 1992 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:55.482718 5134 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.484025 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.485852 4606 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.522185 2136 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:55.522449 4606 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.485680 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.346478 5106 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.487665 5122 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.487846 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.490779 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.493966 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.497351 2614 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:55.500809 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.501434 2649 raft_consensus.cc:1226] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:55.507324 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.543989 5122 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:55.544082 2641 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:55.520459 2646 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:55.477028 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:55.532269 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 59182.5us
W20260924 00:22:55.549568 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 52332.3us
I20260924 00:22:55.552131 2614 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:55.553681 5122 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:55.554108 5122 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.554494 5122 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:55.556957 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.563024 5122 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.565788 2614 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:55.566761 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.570187 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.571017 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:55.571861 5106 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:55.572765 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.575700 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.576380 2140 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
W20260924 00:22:55.577188 1925 debug-util.cc:398] Leaking SignalData structure 0x7b0800333b20 after lost signal to thread 2169
W20260924 00:22:55.578905 1925 debug-util.cc:398] Leaking SignalData structure 0x7b08003b7400 after lost signal to thread 2673
I20260924 00:22:55.819829 5141 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:55.819757 5120 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:55.827095 1925 debug-util.cc:398] Leaking SignalData structure 0x7b08002ad1e0 after lost signal to thread 2675
I20260924 00:22:55.827788 5120 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.827852 5141 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.830178 5141 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.830415 5120 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.603094 5130 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.590610 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.579365 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.737059 5107 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.861548 5107 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.863770 5107 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.861548 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:55.824450 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 253500us
I20260924 00:22:55.856359 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.865712 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:55.866302 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.866803 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.867964 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.868193 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.868803 2064 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:55.735416 5097 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:55.857151 5122 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:55.871757 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.001072 2064 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:56.000869 5097 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.002020 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.003047 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.003973 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.872740 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.872337 5130 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:55.873515 5107 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:56.005033 5107 raft_consensus.cc:686] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:56.005244 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.005908 5107 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.006311 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:56.007155 5099 consensus_meta.cc:287] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.533s user 0.006s sys 0.007s
I20260924 00:22:56.007289 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.876241 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.878101 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.878379 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.008183 2066 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:22:56.007623 5099 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.008399 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.008594 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.009696 2062 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:55.735023 5052 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:55.878842 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.004757 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.889567 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.890342 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.011420 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.011680 2069 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:55.890667 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.012540 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.012984 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.013725 1993 raft_consensus.cc:2379] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:55.891445 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.891659 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.018697 5097 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.019722 1991 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260924 00:22:56.020409 5099 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.892364 2640 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:55.579018 2614 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:56.022957 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 443974us
I20260924 00:22:56.023557 5134 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:56.023929 5134 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:55.902133 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.901312 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.902971 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.902185 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:55.912298 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 341672us
I20260924 00:22:55.908728 2065 raft_consensus.cc:1226] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:55.687314 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.026383 2648 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:56.026355 2110 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:55.924832 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:55.935122 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:56.027511 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 102740us
I20260924 00:22:55.934270 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.028246 2110 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:55.934859 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.029975 5102 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:55.936739 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.030730 5102 consensus_queue.cc:1046] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:56.031533 5119 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:55.936398 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.031762 5102 raft_consensus.cc:3032] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:56.032223 5102 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:22:55.949719 2035 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.016s sys 0.008s
I20260924 00:22:55.629812 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.750100 2613 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:22:56.032742 2035 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.017s sys 0.009s
I20260924 00:22:55.728206 4606 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:55.961090 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.960624 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:56.033658 2642 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:55.952308 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:55.962015 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:55.582564 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.649430 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.628670 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.035326 5102 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:55.590878 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.010221 5052 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.040588 5102 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.040901 5052 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:55.579489 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.023871 2614 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:56.044255 2140 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:22:56.044692 2614 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.046028 2614 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:56.024425 5134 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:56.030030 5140 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:56.049541 5140 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.049907 5140 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:56.051263 5052 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:55.935662 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:55.936668 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.053438 5052 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:56.033154 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 283116us
W20260924 00:22:56.033838 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 73287.9us
I20260924 00:22:55.603466 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.055672 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:56.035925 2643 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:56.057102 5142 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:56.057595 5142 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.057907 1960 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.059267 5099 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:55.615513 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:55.996223 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 196) took 3094 ms. Trace:
I20260924 00:22:56.053413 2143 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:56.037192 1987 raft_consensus.cc:1226] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:56.037585 4606 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:56.038089 5120 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.081522 4606 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.081943 4606 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.081715 5120 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.083906 5120 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.011422 2071 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:22:56.038542 5141 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.060602 1890 rpcz_store.cc:276] 0924 00:22:52.901402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:52.908330 (+ 6928us) service_pool.cc:227] Handling call
0924 00:22:55.996130 (+3087800us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2941696,"reported_tablets":1}
I20260924 00:22:56.036736 5130 raft_consensus.cc:2781] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:56.057934 5142 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.064155 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.065702 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.069271 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.070964 2109 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:56.072082 1993 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10. Ops: []
I20260924 00:22:56.072712 5106 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:56.073199 5122 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:56.073359 5131 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:56.085584 5141 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.060475 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:56.073156 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 44964us
I20260924 00:22:56.092926 1960 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:56.094208 5052 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:56.094626 5052 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.094807 2110 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:56.095007 5052 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:56.091461 5122 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.101568 5122 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:56.104095 2110 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:56.112553 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.116034 1960 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.113466 2110 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.090917 5106 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.120671 5144 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:56.123657 2110 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.122725 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:56.091491 5131 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.105685 5102 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:56.127730 2109 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.126677 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:56.121520 5148 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:22:56.130467 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.121219 5147 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:56.121428 5143 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.179515 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.187256 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.187531 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.134934 4606 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.123921 5106 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.125706 5118 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.120797 5127 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.120576 5099 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:22:56.209558 5146 log.cc:924] Time spent T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260924 00:22:56.180733 2068 consensus_queue.cc:237] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:22:56.189741 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.122455 5130 raft_consensus.cc:686] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
W20260924 00:22:56.182126 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 94040.6us
I20260924 00:22:56.106112 5052 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.125504 5144 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.196651 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.211586 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.215411 5144 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:56.216912 5052 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.218343 5141 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.218394 5130 consensus_queue.cc:237] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.223863 2613 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.226464 5151 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:56.227237 2613 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.228291 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.228636 2613 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.127293 5107 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:56.128661 5102 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.209183 5127 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.231110 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.128400 5131 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.179787 5097 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:56.242772 5152 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:56.243163 5097 raft_consensus.cc:686] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:56.246496 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.233489 4606 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.180284 5148 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:56.181962 5147 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:22:56.249646 5147 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.249584 5148 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:56.125664 5142 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.254174 5142 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.256650 5134 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.258682 5134 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.258838 5148 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.259464 5152 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.259783 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.226841 5151 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.260463 5151 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.260754 5148 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.265659 5147 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.265419 5097 consensus_queue.cc:237] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.267490 5147 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:56.270722 5140 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:56.273241 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 145544us
I20260924 00:22:56.274160 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.279708 2109 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.241732 5102 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:56.234787 5107 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.288435 5148 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260924 00:22:56.292321 5148 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.292696 5148 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Advancing to term 12
W20260924 00:22:56.291572 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 166119us
I20260924 00:22:56.293156 5144 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:56.300693 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 212197us
I20260924 00:22:56.293365 5107 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:56.306769 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
W20260924 00:22:56.305835 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 71268.3us
I20260924 00:22:56.308146 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
W20260924 00:22:56.310075 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 81503.6us
I20260924 00:22:56.311259 2613 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.311982 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.313460 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.321702 5163 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:56.322047 2110 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.324182 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.309537 5140 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.326239 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.294087 5122 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.327075 2613 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.329581 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.324870 5119 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:56.323010 5134 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:56.338721 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.334219 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.323213 5159 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:56.375351 5107 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.332736 5130 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:56.350287 2613 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.353945 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.379693 2613 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.379954 5148 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.357018 5154 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:56.380847 5154 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.381260 5154 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.384572 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.358369 5137 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:22:56.332060 5106 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.385504 5137 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.383600 5160 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.386472 5162 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:56.361213 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.386889 5162 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.387213 2641 raft_consensus.cc:2379] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:56.387301 5162 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:56.330737 5119 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.387935 5160 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:56.388520 5160 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.330840 5099 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:56.388980 5160 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.341782 5134 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.389611 5134 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.390801 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.391585 1993 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:56.393265 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.372997 5102 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.393548 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.331194 5097 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.336300 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.366132 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.397434 2645 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:56.386834 5106 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.331765 5151 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.400203 5151 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.357597 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.365871 5142 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:56.400924 5142 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.400972 5148 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.401525 5142 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.369506 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.405313 5102 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.405726 1987 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 10: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:56.365840 2649 raft_consensus.cc:2379] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 10: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:56.355461 5144 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.357602 5157 consensus_queue.cc:1046] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:56.385938 5137 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:56.417832 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.419080 5149 raft_consensus.cc:3032] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:22:56.419430 5149 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.405439 5154 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.420192 5149 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.338419 5140 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:56.365238 5122 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.421221 5154 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.381865 2613 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.422521 5122 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:56.424429 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.426750 5131 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.427347 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.428436 2613 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.428719 5131 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.429111 5122 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:56.429734 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.432667 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:56.409631 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 85531.8us
I20260924 00:22:56.411476 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.433414 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.433938 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.434547 2140 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.436335 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.436933 2143 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.437662 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.443039 2140 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 9.
I20260924 00:22:56.407697 5107 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.447280 5134 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:56.409581 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 81273.8us
I20260924 00:22:56.415796 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.450676 5142 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:56.416453 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.451361 5140 consensus_queue.cc:1046] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:22:56.388113 5119 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.452147 5140 raft_consensus.cc:3032] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:56.452507 5140 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.452555 5142 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.453270 2109 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.454722 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.455053 5166 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:56.455506 2136 raft_consensus.cc:2379] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 11: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:56.453320 5140 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.330190 5155 consensus_queue.cc:1046] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260924 00:22:56.456597 2645 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:56.456933 5140 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.445866 1890 catalog_manager.cc:5495] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 7, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 7 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:56.457114 2143 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:56.463303 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.463572 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.458983 5134 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.464166 2643 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:56.460423 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.460616 5166 consensus_queue.cc:1046] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20260924 00:22:56.465638 5156 raft_consensus.cc:3032] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:56.465205 1890 catalog_manager.cc:5495] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 7 to 8, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 8 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:56.466075 5156 raft_consensus.cc:729] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.465441 5119 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.466912 5156 consensus_queue.cc:260] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:56.468206 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 144358us
I20260924 00:22:56.468633 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.469149 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.461462 2645 raft_consensus.cc:1226] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:56.461117 5147 consensus_queue.cc:1046] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260924 00:22:56.471654 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.472653 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.462527 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.460988 5155 raft_consensus.cc:3032] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:56.473485 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.473959 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.474248 5156 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:56.473977 5155 raft_consensus.cc:729] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.458081 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.474926 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.475396 5155 consensus_queue.cc:260] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.474267 2641 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:56.476912 5155 raft_consensus.cc:3037] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.477264 5149 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:56.481091 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.481369 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.481978 2145 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:56.477685 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.478317 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.479221 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.483247 2142 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:56.483409 2641 raft_consensus.cc:2379] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:56.484112 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.485009 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.485359 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.457286 5131 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.486619 2649 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:56.413601 5162 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.480058 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.489205 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.490032 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.490728 2648 raft_consensus.cc:2379] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:56.492195 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.493431 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.432392 5137 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.494076 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.460911 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.495426 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.495595 5137 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.496243 2136 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:56.496460 5137 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:56.496851 5137 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.521725 5137 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.522398 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.524432 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.524622 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.499575 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.500329 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.510977 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.525359 5144 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:56.507179 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.529434 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.530143 5144 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.508280 5162 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.494073 2142 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:56.497515 2109 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.515319 2069 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 10.
I20260924 00:22:56.496799 2143 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:56.516052 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.536152 2645 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:56.517048 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.516397 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.517351 2641 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10. Ops: []
I20260924 00:22:56.537153 5140 raft_consensus.cc:2741] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260924 00:22:56.519126 2643 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:56.520572 5167 consensus_queue.cc:1046] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:56.536950 5144 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.538625 5149 raft_consensus.cc:3032] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:56.538976 5149 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.539099 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.539358 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.539860 5142 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:56.540345 5142 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.539752 5149 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.540524 2109 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.541191 5140 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.541044 5149 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.542271 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.542538 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.543354 1987 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:56.544291 5162 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.547775 2643 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:56.497751 5166 consensus_queue.cc:1046] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260924 00:22:56.540736 5142 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:56.551405 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.551796 1987 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:22:56.553514 2137 raft_consensus.cc:1226] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:56.553593 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.559243 1990 raft_consensus.cc:1226] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:56.559659 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.554806 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.564702 2068 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:22:56.566728 1995 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:56.566984 1996 raft_consensus.cc:1226] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:56.569121 5157 consensus_queue.cc:1046] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:22:56.569550 1989 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260924 00:22:56.569911 2035 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.570202 5157 raft_consensus.cc:3032] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:56.570578 5157 raft_consensus.cc:729] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.570637 5167 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:56.571069 2035 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.571410 5157 consensus_queue.cc:260] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.572619 5157 raft_consensus.cc:3037] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.573024 1989 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:56.574162 5162 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:56.574659 2109 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.575767 2109 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.556300 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.576313 5156 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.557121 5140 raft_consensus.cc:3032] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:56.557291 2069 raft_consensus.cc:1261] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:56.576886 5140 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.557171 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.577749 5140 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.578287 2035 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.579160 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.579309 5168 consensus_queue.cc:1046] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:56.579799 5167 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.580123 5141 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:56.580400 5168 raft_consensus.cc:3032] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:56.580757 5168 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.581571 5168 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.579090 5140 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:56.582484 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.575840 5155 raft_consensus.cc:2741] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260924 00:22:56.582888 5168 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.497749 5119 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.555833 5142 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.587347 1992 raft_consensus.cc:1226] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 3. Current term is 4. Ops: []
I20260924 00:22:56.588266 5142 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:56.585599 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.592953 5165 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.593102 5119 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.594625 5119 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.595927 2109 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.596563 5165 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:56.596637 5119 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:56.502473 5160 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.597919 2613 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.599176 5160 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:56.587554 2035 consensus_peers.cc:487] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d -> Peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667): Couldn't send request to peer 9e79063532384ada8ea8f8205478adcf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88e267ffb8c4467b93815c1a9e3d3664. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:22:56.599785 5160 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:22:56.600322 2649 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:56.600442 5160 raft_consensus.cc:686] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:56.600834 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:56.601405 5160 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.604413 1992 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:56.596921 2109 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.605255 5157 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:56.605334 5167 consensus_queue.cc:1046] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:56.606446 5167 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:56.606662 5141 raft_consensus.cc:3032] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:56.607049 5141 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.607239 5167 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 LEADER]: Stepping down as leader of term 10
I20260924 00:22:56.607615 5167 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.608199 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.607811 5141 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:56.608449 5157 consensus_queue.cc:1046] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:56.609099 5141 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.609263 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.609647 5163 raft_consensus.cc:3032] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:56.610050 5163 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.611052 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.610767 5163 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.612147 5163 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:56.608355 5167 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.613658 5167 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:56.614770 1992 raft_consensus.cc:1226] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:56.614871 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.617404 5162 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260924 00:22:56.619381 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.619923 2137 raft_consensus.cc:1226] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:56.620158 5140 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:56.620522 5140 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.620844 5140 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.622157 5162 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.623775 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.624784 5162 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.605619 2109 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.625204 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.625841 5119 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:56.626029 5162 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:56.626686 5119 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.627014 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.627108 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.627573 2142 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.627166 5119 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.628052 5162 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:56.628476 5162 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.628815 5162 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.629946 5119 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.633448 2142 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:56.636060 5162 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.637321 2137 raft_consensus.cc:1226] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 10. Current term is 11. Ops: []
I20260924 00:22:56.636081 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.637948 2142 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:56.637643 5137 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.638654 5162 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.638933 2140 raft_consensus.cc:2379] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:56.637727 1960 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.638909 2110 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.640264 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.640959 5142 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:22:56.641081 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.641511 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.641610 5142 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.641943 2063 raft_consensus.cc:2396] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260924 00:22:56.642697 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.643347 5119 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.599115 2613 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:56.643497 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.644237 5131 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:56.644476 2110 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:22:56.645118 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 46198.4us
I20260924 00:22:56.645627 2110 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.644738 5131 raft_consensus.cc:686] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:56.646368 5119 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.646749 2142 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:56.647043 5131 consensus_queue.cc:237] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.648622 2613 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.649030 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.649421 5154 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:22:56.650621 2613 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.651189 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.652941 2110 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:56.653038 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.651417 5151 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.653906 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.654471 2613 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.651585 5162 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.654937 5165 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:22:56.663551 5151 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.655041 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.655380 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.664083 5166 consensus_queue.cc:1046] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:22:56.655596 5142 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.657125 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:56.652140 5119 raft_consensus.cc:2726] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.664635 5162 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.665190 5166 raft_consensus.cc:3032] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:22:56.665555 5166 raft_consensus.cc:729] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.666272 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.663233 2613 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.668202 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.668395 2613 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.669723 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.673287 5165 raft_consensus.cc:686] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.674631 5165 consensus_queue.cc:237] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.677150 5154 raft_consensus.cc:2781] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:56.677896 5154 raft_consensus.cc:686] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [term 8 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:56.678651 5134 consensus_queue.cc:1046] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:56.679044 5154 consensus_queue.cc:237] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.679869 5122 raft_consensus.cc:3032] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:56.680301 5122 raft_consensus.cc:729] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:56.681602 5102 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.682101 5102 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.682806 5122 consensus_queue.cc:260] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.684265 2143 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:56.667757 5151 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:56.685127 2137 raft_consensus.cc:1226] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:56.684478 5102 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:56.686831 2139 raft_consensus.cc:1226] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:56.686966 5102 consensus_queue.cc:1046] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:56.688773 5159 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:56.689165 5144 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:56.689596 5144 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.689678 5122 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.689970 5144 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:56.689980 5127 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:56.690595 5127 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.691624 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.691761 5127 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.667793 5142 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.692203 2035 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.693601 5149 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:56.694538 2035 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.685096 5151 raft_consensus.cc:686] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 9 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:56.695546 5167 consensus_queue.cc:1046] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 00:22:56.696388 5167 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:56.696344 5151 consensus_queue.cc:237] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.697011 5167 raft_consensus.cc:686] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.697108 5141 raft_consensus.cc:3032] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:56.697495 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:56.697932 5167 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.700578 1960 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.701531 5162 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:56.666297 5166 consensus_queue.cc:260] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.701960 5162 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:56.702383 5162 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:56.702261 5149 consensus_queue.cc:1046] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 00:22:56.702607 5166 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:56.702917 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.703262 5149 raft_consensus.cc:3032] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:56.703636 5149 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.697508 5141 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.704368 5149 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.701439 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.707018 5149 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.704926 5141 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.700850 5127 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.705641 5140 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.708742 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.709443 5142 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:56.710546 5107 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.711252 2136 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:56.714679 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.714754 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.716385 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.716741 2140 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:56.721932 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.723006 5145 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:56.710297 5137 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:56.723881 4606 consensus_queue.cc:1035] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:22:56.723489 5099 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:56.713742 5142 consensus_queue.cc:1046] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260924 00:22:56.725728 2613 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.720990 5141 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:56.731516 5142 raft_consensus.cc:3032] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:56.731986 5142 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:56.733332 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.725625 5052 consensus_queue.cc:1046] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20260924 00:22:56.736035 5052 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 LEADER]: Stepping down as leader of term 10
I20260924 00:22:56.736475 5052 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:56.737516 5052 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.738895 5099 raft_consensus.cc:686] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:56.739140 5052 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:56.739815 5099 consensus_queue.cc:237] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:22:56.740064 5144 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.732924 5142 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.743928 5159 raft_consensus.cc:3032] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:56.744441 5159 raft_consensus.cc:729] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.742810 5142 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:56.745435 5159 consensus_queue.cc:260] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.750264 5159 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.741966 5144 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.751967 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.752290 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.752991 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.753077 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.753815 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.754436 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.772318 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.772953 2143 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:56.777771 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.782775 2140 raft_consensus.cc:1226] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5. Ops: []
I20260924 00:22:56.765717 5134 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.789417 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.791231 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.792125 2137 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:56.793675 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.791865 2144 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:56.800576 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.802187 5140 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:56.804288 5162 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.808772 2139 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:22:56.810067 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.810250 2142 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11. Ops: []
I20260924 00:22:56.812346 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.812986 2141 raft_consensus.cc:2379] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:56.813099 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.814043 2138 raft_consensus.cc:1261] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:22:56.817242 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad"
dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
mode: GRACEFUL
new_leader_uuid: "3eb374c9744b4c7e92dc749aece177dd"
from {username='slave'} at 127.0.0.1:39356
I20260924 00:22:56.820448 2649 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 2. Current term is 3. Ops: []
I20260924 00:22:56.826771 5145 raft_consensus.cc:686] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:56.827402 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.827764 5145 consensus_queue.cc:237] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.833361 2035 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.834622 2035 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.835321 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.837627 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:56.840608 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.841948 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:56.843266 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.843554 2649 raft_consensus.cc:2379] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 9: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:56.843964 2070 raft_consensus.cc:2379] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:56.845108 2069 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260924 00:22:56.845386 5176 log.cc:924] Time spent T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260924 00:22:56.845175 5157 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260924 00:22:56.849843 2141 raft_consensus.cc:1261] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260924 00:22:56.850452 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:56.852806 2144 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:56.851893 2035 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.856389 2068 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:56.838021 2109 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.862190 2109 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:56.863556 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08002a3700 after lost signal to thread 815
W20260924 00:22:56.862608 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 139215us
W20260924 00:22:56.866870 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08003a2940 after lost signal to thread 1928
W20260924 00:22:56.868089 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08003732a0 after lost signal to thread 2020
W20260924 00:22:56.869513 2017 debug-util.cc:398] Leaking SignalData structure 0x7b0800397fa0 after lost signal to thread 2095
W20260924 00:22:56.870687 2017 debug-util.cc:398] Leaking SignalData structure 0x7b080015d4e0 after lost signal to thread 2169
I20260924 00:22:56.869292 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.870918 1890 catalog_manager.cc:5495] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 6 to 7, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 7 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:56.863020 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:22:56.871773 2017 debug-util.cc:398] Leaking SignalData structure 0x7b0800104960 after lost signal to thread 2673
I20260924 00:22:56.872862 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:56.873942 2017 debug-util.cc:398] Leaking SignalData structure 0x7b080032a7c0 after lost signal to thread 2675
I20260924 00:22:56.851198 5163 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260924 00:22:56.864249 5127 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.875082 2142 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:56.875123 2070 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:56.879680 2035 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.876070 5120 consensus_queue.cc:1046] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:22:56.876833 2144 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 9. Current term is 10. Ops: []
I20260924 00:22:56.877637 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.878140 5163 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.872901 1890 catalog_manager.cc:5495] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 9 to 10, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 10 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:56.886984 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.886888 1890 catalog_manager.cc:5495] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 7 to 8, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 8 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:56.889815 1890 catalog_manager.cc:5495] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 6 to 7, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 7 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:56.891242 1890 catalog_manager.cc:5495] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 8 to 9, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 9e79063532384ada8ea8f8205478adcf (127.0.203.4). New cstate: current_term: 9 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:56.880774 2035 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.864878 5157 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.865043 5162 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:56.879509 5156 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:56.896598 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:56.861660 5154 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260924 00:22:56.901329 2138 raft_consensus.cc:595] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Received request to transfer leadership to TS 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.905645 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:56.906502 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.881227 5141 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.908398 2035 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.906730 2138 raft_consensus.cc:603] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:56.910338 5184 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.909938 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.913442 5119 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.906066 5167 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.915812 5149 raft_consensus.cc:3032] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:22:56.916054 2109 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.916263 5149 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:56.916690 1989 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 9. Current term is 10. Ops: []
I20260924 00:22:56.917181 2109 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.917575 1992 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 9. Current term is 10. Ops: []
I20260924 00:22:56.917222 5149 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.918342 5102 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260924 00:22:56.918569 5166 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:22:56.918776 5149 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:56.918959 5102 consensus_queue.cc:1046] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:22:56.920251 5137 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:22:56.920593 5137 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.921378 5137 consensus_queue.cc:260] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.921819 5119 consensus_queue.cc:1046] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:56.921389 5160 consensus_queue.cc:1035] T 2f75d53c3ccf437cac00ba5d83b2c165 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:56.922741 5159 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:22:56.922864 5184 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:56.923172 5159 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.923321 5184 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:56.923758 5184 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:56.895632 5127 consensus_queue.cc:1046] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260924 00:22:56.924052 5159 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.925824 5127 raft_consensus.cc:3032] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:22:56.926152 5159 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:56.927532 5137 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:56.878477 5168 raft_consensus.cc:2932] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:56.924476 2035 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.926189 5127 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.933799 5167 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:56.934542 5127 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.938392 5127 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:56.939510 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.945662 5141 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:56.946795 2035 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.918586 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.949573 2035 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.951679 5155 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260924 00:22:56.953120 5163 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:56.953863 5141 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:56.953089 2035 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.954347 5141 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.954766 5141 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:56.954861 5163 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:56.955238 5163 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.955615 5163 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:56.951504 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.957279 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.960857 5167 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:22:56.960884 5155 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260924 00:22:56.961720 2035 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.962464 5167 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:56.964013 2035 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.958712 2109 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.965698 2109 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:56.967542 2035 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:56.968709 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.971086 2035 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:56.989890 2109 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:56.992937 5147 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:22:56.993610 5147 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:56.994081 5147 raft_consensus.cc:686] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:56.995249 5147 consensus_queue.cc:237] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:56.996923 5166 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:56.997439 5166 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:56.998333 2649 raft_consensus.cc:1226] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9. Ops: []
I20260924 00:22:56.999377 5166 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:57.000607 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:57.000473 1890 catalog_manager.cc:5495] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 10 to 11, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 11 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:22:57.003338 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:57.004648 5165 consensus_queue.cc:1046] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:57.006368 5119 raft_consensus.cc:3032] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:57.006821 5119 raft_consensus.cc:729] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:57.008193 5119 consensus_queue.cc:260] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:57.015533 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:57.015625 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:57.019510 4606 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:22:56.966907 5155 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:22:56.990031 5137 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:57.020460 5166 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:22:57.022548 5119 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:57.026769 1994 raft_consensus.cc:1226] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:56.986512 5120 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:56.986404 5157 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:22:57.032573 1994 raft_consensus.cc:2932] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:57.042798 1989 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
W20260924 00:22:57.047510 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 76534.5us
I20260924 00:22:57.049311 2139 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:57.056427 2035 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:57.059852 2139 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 11. Current term is 12. Ops: []
I20260924 00:22:57.044461 5167 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:57.079061 5167 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:57.095921 2065 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59992
I20260924 00:22:57.097338 2065 raft_consensus.cc:595] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:57.097766 2065 raft_consensus.cc:603] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:57.095146 2144 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:57.098383 1994 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260924 00:22:57.102577 5167 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:57.102389 5097 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:57.118712 5134 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:57.121532 2144 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260924 00:22:57.121272 5150 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:22:57.123801 5141 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.125360 5150 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:22:57.125885 5141 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:57.119290 1893 catalog_manager.cc:5495] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 6 to 7, leader changed from 9e79063532384ada8ea8f8205478adcf (127.0.203.4) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:57.129724 2069 raft_consensus.cc:1226] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:57.122767 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:57.143033 4606 consensus_queue.cc:1046] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:22:57.143173 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:57.096272 5149 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:57.150130 4606 raft_consensus.cc:3032] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:22:57.150543 4606 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:57.151273 4606 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.152391 4606 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:57.152110 5099 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:57.154073 5099 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:22:57.154414 5099 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:57.152412 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:57.155180 5099 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.152926 5052 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260924 00:22:57.172811 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 79015us
I20260924 00:22:57.082989 5184 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:57.189522 5168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:43182 (request call id 13) took 1059 ms. Trace:
I20260924 00:22:57.145779 5097 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:57.158782 5099 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:57.205753 2141 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260924 00:22:57.209586 1989 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260924 00:22:57.198676 5168 rpcz_store.cc:276] 0924 00:22:56.130204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:56.130360 (+ 156us) service_pool.cc:227] Handling call
0924 00:22:56.306007 (+175647us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b4c000c3200. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d2e7f210 00007fb6d2e8ffb2 00007fb6d2e56ac5 00007fb6d2e7d71d 00007fb6d2e7d383 00007fb6d4a35a79 00007fb6d4a358d4 00007fb6d4a9be6e 00007fb6d4ab7acc 00007fb6d4ab8ada 00007fb6d4ab7c8b
0924 00:22:56.310483 (+ 4476us) log.cc:842] Serialized 492 byte log entry
0924 00:22:57.189405 (+878922us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1279,"raft.queue_time_us":3617,"raft.run_cpu_time_us":2350,"raft.run_wall_time_us":3598,"spinlock_wait_cycles":6658304,"thread_start_us":175913,"threads_started":1,"wal-append.queue_time_us":144133}
I20260924 00:22:57.212252 5198 consensus_queue.cc:1046] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:57.213326 5154 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 00:22:57.094686 5195 consensus_queue.cc:1035] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260924 00:22:57.210644 5186 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:22:57.149076 5149 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:22:57.223446 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 127231us
I20260924 00:22:57.241700 2069 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260924 00:22:57.245820 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:22:57.248799 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 92760.6us
I20260924 00:22:57.233896 5149 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:57.264406 1987 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:22:57.264773 4606 consensus_queue.cc:1046] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:57.266086 4606 raft_consensus.cc:3032] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:57.266721 4606 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:57.268033 4606 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.270534 4606 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:57.224113 5147 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260924 00:22:57.277670 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:57.278301 1987 raft_consensus.cc:595] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:57.278666 1987 raft_consensus.cc:603] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:57.280691 2064 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:57.282217 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260924 00:22:57.284067 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:57.284730 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:57.244441 5167 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.292624 5167 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:57.272989 5052 consensus_queue.cc:1046] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260924 00:22:57.294238 5184 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:57.294575 5200 raft_consensus.cc:3032] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:57.294745 5052 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Stepping down as leader of term 11
I20260924 00:22:57.295105 5200 raft_consensus.cc:729] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:57.295190 5052 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:57.295908 5122 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:57.296056 5052 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.296056 5200 consensus_queue.cc:260] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.298889 5200 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:57.300545 5120 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:22:57.276412 5198 consensus_queue.cc:1046] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260924 00:22:57.316599 5052 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:22:57.273561 5107 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260924 00:22:57.317395 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:57.315407 5149 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.320775 2070 raft_consensus.cc:1226] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:22:57.321221 5163 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.322319 2063 raft_consensus.cc:1226] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:22:57.322522 5149 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:57.323333 5201 log.cc:924] Time spent T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260924 00:22:57.326536 2136 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:57.331002 1990 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260924 00:22:57.333591 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:57.335060 2035 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:57.337493 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:57.341922 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:57.350015 5163 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:57.351677 5107 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260924 00:22:57.353938 1942 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 3eb374c9744b4c7e92dc749aece177dd is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } }
I20260924 00:22:57.361971 5198 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:57.362401 5198 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:57.363188 5198 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:57.364641 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 87743.1us
I20260924 00:22:57.370497 5194 consensus_queue.cc:1046] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:57.371897 5200 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:57.375555 2070 raft_consensus.cc:1261] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260924 00:22:57.376046 2070 pending_rounds.cc:76] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 6
I20260924 00:22:57.377345 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:57.350092 2136 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:22:57.381258 2142 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:22:57.376343 2070 pending_rounds.cc:97] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260924 00:22:57.399061 1893 catalog_manager.cc:5495] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 5, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 5 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:57.401858 1893 catalog_manager.cc:5495] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 7 to 8, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 8 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:57.404747 2035 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:57.411650 2070 raft_consensus.cc:2866] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:22:57.411548 4606 consensus_queue.cc:1046] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:57.412531 4606 raft_consensus.cc:3032] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:22:57.412815 4606 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:57.413487 4606 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.414602 4606 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:57.425762 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:57.427548 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:57.374475 5198 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:57.438573 5166 consensus_queue.cc:1035] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260924 00:22:57.450851 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:57.409345 5149 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:57.461794 2036 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:22:57.465071 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 195586us
I20260924 00:22:57.467262 4606 consensus_queue.cc:1035] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 00:22:57.464663 1996 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:57.464427 5149 raft_consensus.cc:686] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:57.472743 2064 raft_consensus.cc:1261] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260924 00:22:57.478400 2036 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:57.476302 1989 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:57.481503 2071 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260924 00:22:57.472642 5149 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:57.492627 5199 raft_consensus.cc:3032] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:22:57.493121 5199 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:57.493240 5119 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 00:22:57.493919 2068 raft_consensus.cc:1261] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:57.494112 5199 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.496220 5199 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:57.421808 5099 consensus_queue.cc:1046] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20260924 00:22:57.497505 5099 raft_consensus.cc:3032] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:22:57.497866 5099 raft_consensus.cc:729] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
W20260924 00:22:57.498924 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet cfca69df981d4ab89f1ea62a1d8c1557, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.498633 5099 consensus_queue.cc:260] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:57.499328 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet cfca69df981d4ab89f1ea62a1d8c1557, TS 948356bf66a941ef8249ecbd0ec94f6d -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.499933 5099 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:57.473167 5167 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260924 00:22:57.514237 2142 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:57.516306 2066 raft_consensus.cc:1261] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260924 00:22:57.521950 5144 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:57.484524 5141 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:57.527375 2145 raft_consensus.cc:2932] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } } }
I20260924 00:22:57.543690 2036 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:57.542299 5141 raft_consensus.cc:686] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:57.552021 5192 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:57.583938 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:57.591894 2643 raft_consensus.cc:2379] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
W20260924 00:22:57.594693 5206 log.cc:924] Time spent T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260924 00:22:57.583319 5192 raft_consensus.cc:686] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:57.602108 5192 consensus_queue.cc:237] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.605499 5203 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:57.606317 5203 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:57.607407 5203 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.609788 5141 consensus_queue.cc:237] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:57.644971 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 102588us
W20260924 00:22:57.647104 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 97009.6us
I20260924 00:22:57.657403 1890 catalog_manager.cc:5495] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: config changed from index -1 to 5, NON_VOTER 9e79063532384ada8ea8f8205478adcf (127.0.203.4) added. New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:57.665333 2643 raft_consensus.cc:1261] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260924 00:22:57.683372 5119 consensus_queue.cc:1035] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260924 00:22:57.685467 5226 consensus_queue.cc:1035] T 9ff940e15e1849efb194c908520c212f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260924 00:22:57.683890 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 132594us
W20260924 00:22:57.695982 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 85bede790e49439d97e8031810eee902, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:57.696326 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 85bede790e49439d97e8031810eee902, TS 8c129c01e1c047628f6a87b001aa7258 -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:57.754354 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 50909.5us
I20260924 00:22:57.771234 5226 consensus_queue.cc:1035] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
W20260924 00:22:57.815228 5217 log.cc:924] Time spent T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20260924 00:22:57.819332 2641 raft_consensus.cc:1261] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260924 00:22:57.796144 5221 consensus_queue.cc:1035] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.101s
I20260924 00:22:57.842741 5218 ts_tablet_manager.cc:927] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:57.915989 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 832ee611b00d4a42b3245beb345af7ad, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:57.922595 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 832ee611b00d4a42b3245beb345af7ad, TS 3eb374c9744b4c7e92dc749aece177dd -> TS 9e79063532384ada8ea8f8205478adcf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:57.924013 1892 catalog_manager.cc:5495] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 8 to 9, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3). New cstate: current_term: 9 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:57.934860 2067 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:22:57.984808 5162 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260924 00:22:57.998699 5163 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:57.999815 5163 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.022532 2137 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:58.022943 5097 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.023778 5097 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.029292 5223 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.030019 5223 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.042654 5147 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.043428 5147 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.043947 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.044961 2145 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:58.045156 5120 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:22:58.049476 5097 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.055821 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.060086 2068 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:58.034664 5223 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.074743 5163 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.075443 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:58.077373 5202 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:58.077838 5202 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.078184 5202 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:58.079172 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.080188 2138 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
I20260924 00:22:58.079186 5147 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.083513 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.084534 1987 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:58.086786 1988 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260924 00:22:58.087961 5228 log.cc:924] Time spent T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20260924 00:22:58.088261 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:58.088786 1995 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:58.089607 5147 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:58.090090 5147 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.090502 5147 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
W20260924 00:22:58.090762 5229 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.246s user 0.000s sys 0.001s
I20260924 00:22:58.085402 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.093744 2066 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
W20260924 00:22:58.094070 5231 log.cc:924] Time spent T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.251s user 0.000s sys 0.002s
I20260924 00:22:58.094159 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:58.097194 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 298303us
I20260924 00:22:58.112905 5142 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260924 00:22:58.113396 5142 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.113852 5142 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:22:58.113215 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.114637 1995 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:22:58.126828 5155 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.127624 5155 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:58.086373 5227 log.cc:924] Time spent T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.244s user 0.002s sys 0.000s
I20260924 00:22:58.135924 2036 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:58.136407 5163 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.137507 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.137265 5163 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.142985 5141 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:22:58.143550 5155 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.169601 5192 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.170073 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.162297 5195 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:58.170958 5195 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.170449 5192 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.171312 5195 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:58.173458 5155 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.174099 5198 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.174041 5155 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.175268 5203 consensus_queue.cc:1035] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:22:58.175165 5198 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.176765 5155 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.178910 5163 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.182106 5150 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.149765 5120 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260924 00:22:58.171139 1995 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:22:58.172885 5192 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.191393 5150 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.193028 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.196591 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.200237 5186 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.206881 2137 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:58.206974 2649 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:58.207356 5186 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.207944 5150 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.208945 2109 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.209749 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.210062 5137 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:58.210474 5137 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.210836 5137 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:58.212147 2035 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:58.202507 2110 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:58.213217 5163 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.213526 5166 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:22:58.213620 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.214123 5166 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.214531 5166 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:58.210126 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.216063 2066 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:58.216557 2141 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:58.217921 5150 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.213836 5163 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.218794 5150 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.219751 5120 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:58.220197 5163 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.220211 5120 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.221163 5120 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:58.217938 5198 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.212545 5186 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.212160 5145 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.224295 5145 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.224944 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.225710 1995 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:58.226495 5145 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.226956 2614 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.227947 5221 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:58.228335 5221 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.228662 5221 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:58.235590 5202 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.237720 5202 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.237939 5226 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:58.238564 5226 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.240037 5127 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.240660 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.240571 5127 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.246668 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.248055 2068 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:58.248760 2141 raft_consensus.cc:1261] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260924 00:22:58.249727 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.250110 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.250568 2138 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:58.251140 2641 raft_consensus.cc:2379] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:58.240703 5226 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.254431 5127 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.255296 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.256093 2649 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:22:58.256354 2641 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260924 00:22:58.260701 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.261691 1988 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:22:58.258690 5147 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.264046 2110 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:58.264873 2145 raft_consensus.cc:2354] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8.
I20260924 00:22:58.265498 2142 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:58.266513 5159 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:22:58.266921 5159 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.268352 5159 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:58.268821 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.269801 1995 raft_consensus.cc:2379] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:58.273736 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.274331 2641 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:22:58.277081 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.278018 1994 raft_consensus.cc:2379] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:58.277842 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.278421 1996 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:58.279394 5233 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.279474 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:58.280186 1990 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:58.280066 5233 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.282177 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.295832 2142 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:58.297312 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.298571 5145 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:58.298980 5145 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.299331 5145 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:58.299527 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:58.300288 2068 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:58.300927 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:58.302069 5240 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:22:58.302472 5240 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.302929 5240 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:22:58.309648 5242 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.310277 5242 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.312464 5242 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.313145 5155 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:58.315551 5155 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.317903 5155 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.314461 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.314478 5147 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.321421 5233 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.322494 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.329715 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.331890 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.330636 1990 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:58.333400 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:58.333851 2142 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:58.337361 2641 raft_consensus.cc:2379] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:58.336566 2035 leader_election.cc:398] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8.
I20260924 00:22:58.341476 2035 leader_election.cc:401] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:58.337286 2068 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:58.295697 5150 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:58.349606 5246 log.cc:924] Time spent T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260924 00:22:58.353453 1893 catalog_manager.cc:5495] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 5 to 6, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } } }
I20260924 00:22:58.357170 5159 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.358945 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:58.363325 5186 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:22:58.363826 5186 raft_consensus.cc:686] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:58.364894 5186 consensus_queue.cc:237] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.289914 5192 consensus_queue.cc:1035] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260924 00:22:58.367524 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.369274 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.369972 2645 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:22:58.370854 2145 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:22:58.340504 5240 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.373625 5240 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.375950 2649 raft_consensus.cc:2379] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 9: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
W20260924 00:22:58.378372 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 90700.8us
I20260924 00:22:58.378882 5199 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 8
I20260924 00:22:58.380676 1994 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:58.381546 5194 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:58.382025 5194 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.382061 5192 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:22:58.383922 5192 consensus_queue.cc:1035] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:22:58.384352 5194 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.385267 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:58.386278 2649 raft_consensus.cc:2379] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 3: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:58.398630 5221 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.380441 5198 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.408300 2110 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:58.414069 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 44950.5us
I20260924 00:22:58.406443 5218 tablet_copy_client.cc:316] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.2:39591
I20260924 00:22:58.418155 2142 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:58.423480 1987 raft_consensus.cc:1261] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:22:58.425997 5198 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.427116 5120 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.439266 5198 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.427183 5166 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.440285 5159 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.426447 5137 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.441437 5166 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.443060 5242 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.442751 5142 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.443575 5242 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.448967 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:58.465346 2066 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 12
W20260924 00:22:58.447106 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 147252us
I20260924 00:22:58.466879 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:58.473774 1995 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:58.476645 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.477486 2139 raft_consensus.cc:2379] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:58.447641 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:58.478369 1989 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:58.478777 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:58.480252 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.480895 1988 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 12.
I20260924 00:22:58.481074 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.478830 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.481729 2070 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:22:58.452246 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:58.446417 5137 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.496124 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.491268 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.445041 5142 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.491910 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:58.439303 5155 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:58.439271 5120 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.497995 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:58.504757 2142 raft_consensus.cc:2379] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 12: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:58.505268 2649 raft_consensus.cc:2379] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:22:58.505481 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:58.505714 5155 raft_consensus.cc:686] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:58.506014 5119 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.511559 2144 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:22:58.514758 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.511755 2137 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:58.525333 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.526115 1996 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:58.523315 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:58.528689 1994 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 8
I20260924 00:22:58.528815 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:58.532228 2614 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.425568 5195 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.530040 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.534269 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.519594 5163 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:58.497125 5198 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260924 00:22:58.519770 5192 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.538787 2137 raft_consensus.cc:2354] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:58.529639 5194 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:58.556423 2070 raft_consensus.cc:2354] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5.
I20260924 00:22:58.556505 2035 leader_election.cc:398] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
I20260924 00:22:58.557011 2035 leader_election.cc:401] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:58.536500 5186 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:58.532552 5221 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.559026 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:58.534914 5242 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.562134 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:58.562649 5154 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:58.563035 5155 consensus_queue.cc:237] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.585996 2143 raft_consensus.cc:2354] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7.
I20260924 00:22:58.534770 5119 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.567771 5199 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8.
I20260924 00:22:58.569453 5224 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.570297 5141 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:58.561653 1995 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:22:58.553092 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:58.556303 5194 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.587476 2144 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 12.
I20260924 00:22:58.597160 2142 raft_consensus.cc:3037] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:58.579151 5186 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.593775 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.593905 5166 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:58.620029 2035 leader_election.cc:398] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7.
I20260924 00:22:58.623272 2035 leader_election.cc:401] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260924 00:22:58.559655 5192 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.581832 5154 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.594393 5163 raft_consensus.cc:3037] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 5
I20260924 00:22:58.624675 5141 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.619798 5224 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.636737 2110 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:58.642623 5119 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.643179 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:58.645648 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.646759 2081 tablet_copy_service.cc:140] P 948356bf66a941ef8249ecbd0ec94f6d: Received BeginTabletCopySession request for tablet 88e267ffb8c4467b93815c1a9e3d3664 from peer 9e79063532384ada8ea8f8205478adcf ({username='slave'} at 127.0.0.1:43194)
I20260924 00:22:58.649170 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.649209 5137 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.649662 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:58.653400 5199 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:58.656522 2142 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:58.662024 5195 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.678620 2071 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:58.666765 5145 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.627393 1891 catalog_manager.cc:5495] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 5 to 6, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 6 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:58.668818 5154 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:58.682420 1891 catalog_manager.cc:5495] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 8 to 9, leader changed from 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 9 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:58.672420 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.622843 5194 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:58.623453 5186 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:58.624254 2645 raft_consensus.cc:3037] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:58.675092 1994 raft_consensus.cc:2454] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:22:58.678079 2613 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.680943 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:58.686969 2641 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 5
I20260924 00:22:58.683866 2066 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 12.
I20260924 00:22:58.680212 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.683915 1891 catalog_manager.cc:5495] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d reported cstate change: term changed from 6 to 7, leader changed from 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1) to 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2). New cstate: current_term: 7 leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:58.690783 5225 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.691560 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:22:58.691455 5225 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.692401 2648 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:58.693351 4606 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.687922 2140 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 9.
I20260924 00:22:58.676779 5147 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:22:58.686556 1995 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:58.695700 2109 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.679097 5240 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader election won for term 12
I20260924 00:22:58.707367 2138 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:22:58.680343 2081 tablet_copy_service.cc:161] P 948356bf66a941ef8249ecbd0ec94f6d: Beginning new tablet copy session on tablet 88e267ffb8c4467b93815c1a9e3d3664 from peer 9e79063532384ada8ea8f8205478adcf at {username='slave'} at 127.0.0.1:43194: session id = 9e79063532384ada8ea8f8205478adcf-88e267ffb8c4467b93815c1a9e3d3664
I20260924 00:22:58.678889 5166 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.691367 5192 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.716223 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.706238 4606 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.718721 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:58.721203 5142 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.721771 5142 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.722561 5225 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.723640 5142 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.724143 5240 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 12 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:58.721196 5166 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.725030 5240 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.726732 5145 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.725519 5257 raft_consensus.cc:2781] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:58.730270 5257 raft_consensus.cc:686] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [term 8 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:58.730858 5119 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:58.731396 5119 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.731252 5257 consensus_queue.cc:237] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.733661 5119 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.735630 1989 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:58.739251 2645 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:58.740341 2641 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:58.741621 2035 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.742816 5167 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:58.743328 5167 raft_consensus.cc:686] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:22:58.744223 5167 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.755105 2070 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:58.755921 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:58.708076 5159 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:58.756017 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:58.760582 2109 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:58.761576 2071 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 11
I20260924 00:22:58.763147 5198 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:58.698998 5137 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.774159 4606 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.764278 5159 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.783916 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:58.787541 4606 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.770407 5224 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.761651 5186 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:58.810200 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 49173.4us
I20260924 00:22:58.785290 5159 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:58.790048 5155 raft_consensus.cc:2781] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:22:58.810981 4606 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.816519 5150 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.817453 5137 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.820542 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:58.786581 5119 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.790212 5147 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:58.780683 5198 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.854343 1892 catalog_manager.cc:5495] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:22:58.869100 2066 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:58.829535 5195 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:22:58.884202 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 69410.6us
I20260924 00:22:58.898033 2110 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:58.883786 5155 raft_consensus.cc:686] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:58.899596 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.899509 5155 consensus_queue.cc:237] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.900405 1989 raft_consensus.cc:2379] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:58.902266 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.905988 1989 raft_consensus.cc:2379] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:58.861369 5166 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:58.906724 4606 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.889081 5127 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.903272 5141 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.938175 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:58.952329 1995 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:22:58.956911 2110 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:22:58.964133 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 256428us
I20260924 00:22:58.883854 5119 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.869975 5150 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.968225 5127 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.975732 5150 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.976488 5127 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.956283 5166 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.977578 5194 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:58.978725 5166 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:58.969673 5264 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:22:58.979532 5264 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:58.979641 5194 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.979911 5264 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:58.980244 5194 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:22:58.951435 5261 raft_consensus.cc:2781] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:22:58.981252 5186 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:22:58.981518 5261 raft_consensus.cc:686] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:58.997118 2071 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:22:58.909405 5195 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.017791 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.018600 1995 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:59.019912 2614 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:59.016547 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.021478 5186 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:58.956463 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:59.022378 1994 raft_consensus.cc:595] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:59.022089 5186 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.024091 5186 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.023218 5154 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:59.024785 5198 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:58.982434 5261 consensus_queue.cc:237] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.027760 5154 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.028887 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:59.029654 2645 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:59.034617 5119 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:59.040172 2081 tablet_copy_source_session.cc:205] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:22:58.937655 5147 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:58.980830 5194 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.050374 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.051124 1994 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:59.055840 2064 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:59.055747 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:59.058466 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 187455us
I20260924 00:22:59.056566 2140 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:22:59.076581 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.077229 2071 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:59.054901 5224 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:22:59.084789 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.086331 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.081386 1994 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 10.
I20260924 00:22:59.086956 2145 raft_consensus.cc:3037] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:59.086831 2071 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:59.092339 2614 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:59.099231 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:59.100759 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:59.102852 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:59.104133 1994 raft_consensus.cc:595] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:59.104657 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:59.105501 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 148671us
I20260924 00:22:59.107420 2066 raft_consensus.cc:2379] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:59.108378 2140 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 9: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:59.086664 5224 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.111454 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.112314 2138 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 10: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:59.114735 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.115154 5195 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.115314 1989 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:22:59.121417 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.122046 1995 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:59.124423 5186 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:59.124913 5186 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.111445 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.125648 5207 raft_consensus.cc:2781] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:22:59.126154 2071 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:59.125675 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:59.126718 5224 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:59.126537 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd66a46f82114470b22385448fd57d51"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:59.116923 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:59.129491 1987 raft_consensus.cc:595] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:59.129948 5271 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:22:59.130489 5271 raft_consensus.cc:686] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:59.131146 5194 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.131455 5271 consensus_queue.cc:237] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:59.132432 5268 consensus_queue.cc:1035] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260924 00:22:59.135051 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:59.135766 1987 raft_consensus.cc:595] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:59.136094 1987 raft_consensus.cc:603] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:22:59.140797 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2730c02ac0a4812841172254e284b90"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:22:59.141510 1987 raft_consensus.cc:595] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:59.142701 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260924 00:22:59.103758 5155 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:59.153208 5194 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260924 00:22:59.149027 5155 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.153668 5194 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:59.154084 5194 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:22:59.129676 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:22:59.172794 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 73669.9us
W20260924 00:22:59.181524 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 84834.7us
I20260924 00:22:59.164788 1996 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
W20260924 00:22:59.187700 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 70857.1us
I20260924 00:22:59.200974 2071 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:22:59.188907 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:22:59.212327 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.213160 2140 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
I20260924 00:22:59.212448 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.218349 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.213034 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.214849 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.223192 2138 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:22:59.222183 2139 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:59.188225 5207 raft_consensus.cc:686] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:59.231526 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.232614 2071 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:59.236421 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.237110 2071 raft_consensus.cc:2379] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 6: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:22:59.248844 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:22:59.250880 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.251782 2071 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
W20260924 00:22:59.254916 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 158480us
I20260924 00:22:59.255086 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.254459 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:22:59.261227 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 169014us
I20260924 00:22:59.262107 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.262720 2071 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:22:59.255569 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.255678 2143 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:59.261173 2064 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:59.256544 2109 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:59.266741 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.268149 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.235029 1890 catalog_manager.cc:5495] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "9e79063532384ada8ea8f8205478adcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:22:59.268718 1987 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:22:59.269858 1994 raft_consensus.cc:2354] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:22:59.270033 1989 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:22:59.270989 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.267769 2613 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:59.260897 5202 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:59.280596 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.281508 2139 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260924 00:22:59.271498 5163 consensus_meta.cc:287] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.601s user 0.009s sys 0.002s
I20260924 00:22:59.284418 1994 raft_consensus.cc:2379] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:59.282246 5160 raft_consensus.cc:2741] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260924 00:22:59.286911 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.287590 2069 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:59.291865 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.292521 1993 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:22:59.297364 5155 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:59.297405 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.276154 5166 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:59.298194 1995 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:22:59.298207 5166 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:59.299288 2645 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:59.299340 5166 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:59.300997 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.301626 2139 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:22:59.304193 5264 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.305375 2136 raft_consensus.cc:2354] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:22:59.306149 5186 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.307020 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:22:59.307358 5194 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.308609 5224 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:59.310361 5207 consensus_queue.cc:237] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:59.329365 2036 leader_election.cc:398] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:22:59.329715 2036 leader_election.cc:401] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:59.325021 2145 raft_consensus.cc:2454] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 10.
I20260924 00:22:59.331553 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.333398 2036 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:59.332165 1994 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:22:59.345398 2613 leader_election.cc:398] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5.
I20260924 00:22:59.345695 2613 leader_election.cc:401] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:59.345844 2036 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:22:59.347572 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:59.348785 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:59.329568 1987 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 5.
I20260924 00:22:59.351562 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.328691 5186 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:22:59.312949 5163 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 4. Current term is 5.
W20260924 00:22:59.331226 5256 log.cc:924] Time spent T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.054s user 0.000s sys 0.003s
I20260924 00:22:59.356627 1996 raft_consensus.cc:2379] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 10: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:59.278422 5242 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260924 00:22:59.349509 5141 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:59.340758 5273 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:22:59.362040 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:59.373809 2036 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:59.380177 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.381057 2069 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:59.352360 5225 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:22:59.395165 2139 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260924 00:22:59.339638 5202 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.364204 5141 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:59.364508 5186 raft_consensus.cc:686] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:22:59.351135 5166 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.425067 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:22:59.404804 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 134308us
I20260924 00:22:59.392248 5225 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.392840 5240 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:22:59.397302 5280 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260924 00:22:59.409677 5199 consensus_meta.cc:287] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.729s user 0.017s sys 0.009s
W20260924 00:22:59.455073 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 109790us
I20260924 00:22:59.427791 5240 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.457864 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:59.422808 5141 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:59.422283 5223 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:22:59.461400 2139 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:59.426915 5225 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:59.466254 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.482582 5223 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.464097 5240 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:59.484668 5223 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.486280 5166 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:59.466840 5277 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:59.488152 5194 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.488317 5277 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:59.488781 5277 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:59.488860 2109 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:59.455063 5260 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260924 00:22:59.490496 2109 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:59.490938 5225 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.491947 5155 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:22:59.492343 5155 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:59.492708 5155 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:22:59.492851 5225 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:22:59.495683 5159 consensus_meta.cc:287] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.643s user 0.010s sys 0.004s
I20260924 00:22:59.496217 5159 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:59.438324 5280 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.466740 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:22:59.497749 5280 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:59.498061 5119 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:59.498226 1993 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 12.
I20260924 00:22:59.498502 5119 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:59.498967 5119 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:59.455768 5199 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7.
I20260924 00:22:59.503495 2613 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:59.468014 5186 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:59.468796 5143 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:59.469413 5122 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5.
I20260924 00:22:59.469980 5202 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.472817 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:22:59.477876 5284 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260924 00:22:59.478703 5224 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.481969 5264 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:22:59.465330 2110 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:59.515762 5271 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:59.520092 5240 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.523692 5277 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.524283 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:22:59.526125 5218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e267ffb8c4467b93815c1a9e3d3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:22:59.526713 5282 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:22:59.564961 5282 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:22:59.565590 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:59.566654 2641 raft_consensus.cc:2379] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:59.531834 5119 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:59.547180 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 74439.5us
W20260924 00:22:59.548441 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 83197.3us
I20260924 00:22:59.549299 5278 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:22:59.517416 2063 raft_consensus.cc:2454] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:22:59.581734 2110 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:59.583010 2110 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:59.549639 5141 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.532882 5143 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.556107 5159 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:22:59.560425 5281 log.cc:924] Time spent T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.266s user 0.003s sys 0.000s
I20260924 00:22:59.590777 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.598837 2144 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:22:59.559199 5271 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.557562 5202 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:22:59.594039 2110 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:59.592857 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:22:59.609584 4606 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:22:59.610248 4606 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.610608 4606 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:59.615115 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.615965 1987 raft_consensus.cc:1261] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:59.616117 1989 raft_consensus.cc:2379] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 11: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:22:59.580974 5278 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:22:59.601013 5143 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:59.618892 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:22:59.605578 5271 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:59.605188 5127 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:22:59.605772 5202 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.630580 2144 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:22:59.617702 5147 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260924 00:22:59.611699 5240 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.622810 5278 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:22:59.636458 1987 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260924 00:22:59.638509 5233 raft_consensus.cc:979] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:22:59.634609 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.646811 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.647599 1987 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:59.630502 5127 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.635993 5202 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:22:59.635299 5147 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.639030 5272 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:22:59.655716 5223 raft_consensus.cc:1067] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
W20260924 00:22:59.669823 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 57462.9us
I20260924 00:22:59.678177 5270 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:22:59.661396 5127 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:22:59.682765 2067 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:22:59.665076 5147 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Advancing to term 13
I20260924 00:22:59.670464 5272 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:22:59.686620 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:22:59.685547 2613 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:22:59.690356 4606 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:59.690825 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 98051.7us
I20260924 00:22:59.694615 1989 raft_consensus.cc:2354] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9.
I20260924 00:22:59.696503 4606 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.697279 1987 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:22:59.702666 1996 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:59.704671 5283 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:22:59.705564 5285 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260924 00:22:59.706959 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.723539 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47770 (request call id 2245) took 1002 ms. Trace:
I20260924 00:22:59.723966 2067 rpcz_store.cc:276] 0924 00:22:58.750581 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:58.755735 (+ 5154us) service_pool.cc:227] Handling call
0924 00:22:59.723382 (+967647us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54003f84d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:22:59.723479 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1697938688}
I20260924 00:22:59.725392 5119 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:22:59.728150 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 236540us
I20260924 00:22:59.737932 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.738555 2063 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 12.
I20260924 00:22:59.739279 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:59.739869 2071 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 11
I20260924 00:22:59.740916 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:22:59.689003 5272 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:59.741611 2064 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:59.743178 2110 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:22:59.744143 2069 raft_consensus.cc:1261] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260924 00:22:59.746994 2067 raft_consensus.cc:1261] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:22:59.752368 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:22:59.688545 5137 consensus_queue.cc:1035] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260924 00:22:59.753026 2649 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:59.753552 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:59.755625 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.803927 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:22:59.805336 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 57582.7us
I20260924 00:22:59.805313 2064 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:22:59.779700 4606 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:22:59.804143 2070 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:22:59.806684 2035 leader_election.cc:398] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:22:59.807000 2035 leader_election.cc:401] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:22:59.808422 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.809258 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.810022 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:22:59.810767 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:59.811388 2066 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260924 00:22:59.807065 4606 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.810153 2068 raft_consensus.cc:1261] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260924 00:22:59.813760 2071 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:22:59.812333 2067 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:22:59.817317 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.809734 5280 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.818066 2142 raft_consensus.cc:2379] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 11: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:22:59.821110 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.742532 5283 raft_consensus.cc:686] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:22:59.830348 2110 leader_election.cc:398] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9.
I20260924 00:22:59.830687 2110 leader_election.cc:401] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260924 00:22:59.829797 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:22:59.804800 5261 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:22:59.742700 5277 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.845541 4606 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.850560 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:22:59.753881 5141 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.868355 2069 raft_consensus.cc:2354] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:22:59.819293 5167 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
W20260924 00:22:59.908191 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 57254us
I20260924 00:22:59.820076 5155 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:22:59.913964 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 93384.7us
W20260924 00:22:59.921401 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 83971.8us
I20260924 00:22:59.924993 5143 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:22:59.927021 5143 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:22:59.905784 5261 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:22:59.910116 5119 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:22:59.932919 2139 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:22:59.942965 5283 consensus_queue.cc:237] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:22:59.943145 5261 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:22:59.966521 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 123074us
I20260924 00:22:59.932528 2645 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:22:59.922837 5223 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260924 00:22:59.935892 5137 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260924 00:22:59.940588 5143 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260924 00:23:00.015719 5268 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:00.016347 5268 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:00.021989 2064 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
I20260924 00:23:00.025583 5141 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260924 00:23:00.036098 5303 log.cc:924] Time spent T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260924 00:23:00.017808 5278 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.039976 5278 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:23:00.022315 5296 log.cc:924] Time spent T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.304s user 0.000s sys 0.001s
I20260924 00:23:00.042840 5155 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:00.013748 5280 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:00.006294 5264 consensus_queue.cc:1035] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260924 00:23:00.006714 5166 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
W20260924 00:23:00.070140 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 97336.7us
I20260924 00:23:00.030090 5159 consensus_queue.cc:1035] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260924 00:23:00.079146 2649 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
W20260924 00:23:00.120555 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 144513us
I20260924 00:23:00.125028 1993 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:23:00.125043 2109 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:23:00.126295 5311 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:23:00.126847 5311 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:00.131440 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.132143 1987 raft_consensus.cc:2354] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10.
I20260924 00:23:00.127825 5311 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.133247 5313 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:23:00.134354 5225 raft_consensus.cc:979] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:23:00.134902 5225 raft_consensus.cc:1067] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:23:00.134768 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.135671 2064 raft_consensus.cc:2379] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 6: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:00.135936 5271 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.137424 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.138284 5271 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:00.140429 1994 raft_consensus.cc:2354] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:23:00.141987 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.142729 1991 raft_consensus.cc:2354] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:00.147527 5266 raft_consensus.cc:979] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:23:00.148092 5266 raft_consensus.cc:1067] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:23:00.148623 5127 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.150892 5127 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:00.151567 2142 raft_consensus.cc:2354] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10.
I20260924 00:23:00.152328 5268 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:00.153160 5268 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:00.153666 5268 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:00.155563 5268 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:00.162153 2064 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:00.165934 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.166488 2064 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:00.075408 5202 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.241923 2648 raft_consensus.cc:1261] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:23:00.171072 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.224354 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:00.176177 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.222384 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.179747 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:00.169085 2109 leader_election.cc:398] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:23:00.296540 2069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd66a46f82114470b22385448fd57d51"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:23:00.326300 2109 leader_election.cc:401] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260924 00:23:00.326550 2069 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:23:00.326956 2069 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:00.297725 2141 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:00.242728 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:23:00.243781 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.328387 2068 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:23:00.328497 2066 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:00.254040 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.324671 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:00.328850 2068 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:00.237006 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.330132 1995 raft_consensus.cc:2379] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 7: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:00.330528 1994 raft_consensus.cc:2379] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:00.298033 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.245512 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:00.179066 5295 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.349s user 0.002s sys 0.000s
I20260924 00:23:00.174731 2064 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:00.179363 5120 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:00.342804 2110 leader_election.cc:398] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
W20260924 00:23:00.249089 5291 log.cc:924] Time spent T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.610s user 0.002s sys 0.000s
I20260924 00:23:00.349488 2064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:23:00.350042 2064 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:23:00.350345 2064 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:00.350675 2641 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:23:00.345891 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.291360 5119 raft_consensus.cc:2726] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9.
I20260924 00:23:00.350381 2140 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []
I20260924 00:23:00.354741 5141 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:00.371805 1989 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 12.
I20260924 00:23:00.354895 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.355608 2036 leader_election.cc:398] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10.
I20260924 00:23:00.375157 2036 leader_election.cc:401] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260924 00:23:00.356071 5163 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:23:00.356717 5218 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45594 (request call id 599) took 2842 ms. Trace:
I20260924 00:23:00.378494 2036 leader_election.cc:398] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:00.378923 2036 leader_election.cc:401] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260924 00:23:00.381063 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.382236 1992 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 10: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:00.357151 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:00.357741 5120 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:00.176391 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:00.354765 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.397606 2062 raft_consensus.cc:2379] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:00.364921 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33232 (request call id 294) took 1745 ms. Trace:
I20260924 00:23:00.375514 1996 raft_consensus.cc:2354] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:23:00.374961 2141 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:23:00.396193 2137 raft_consensus.cc:2379] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 10: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:00.386030 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:00.384277 2069 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.383193 2064 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.383330 2068 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.400977 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:00.402923 2064 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:00.404672 2068 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:00.380460 5272 consensus_meta.cc:287] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.612s user 0.010s sys 0.000s
I20260924 00:23:00.409893 2069 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:00.179141 5147 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:00.353125 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 188071us
I20260924 00:23:00.390301 5268 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10.
I20260924 00:23:00.416057 1960 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:00.390223 5260 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:00.420135 2066 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:23:00.403371 5166 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:23:00.410074 5186 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:00.431756 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:00.428717 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.432680 2144 raft_consensus.cc:2379] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:00.429734 2136 raft_consensus.cc:2354] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 12. Current term is 13.
I20260924 00:23:00.433300 1996 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260924 00:23:00.426352 2142 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:23:00.417001 5318 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260924 00:23:00.389933 5141 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:00.449918 2137 raft_consensus.cc:1261] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260924 00:23:00.427340 5270 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:00.415172 5223 consensus_queue.cc:1035] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260924 00:23:00.429246 5266 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:23:00.455380 2137 raft_consensus.cc:1261] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:23:00.438886 5186 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:00.460577 1960 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:00.447712 2644 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:23:00.462148 5202 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:00.414163 5272 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.462746 5266 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:00.461277 5186 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:00.397981 1892 rpcz_store.cc:276] 0924 00:22:58.619006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:58.622882 (+ 3876us) service_pool.cc:227] Handling call
0924 00:23:00.364832 (+1741950us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:58.910605 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:59.150129 (+239524us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b5c0028fd0c. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6dae45501 00007fb6dae453f1 00007fb6dae4012c 00007fb6d84dfa59 00007fb6d84cb603 00007fb6d84cb2b8 00007fb6d84cfba1 00007fb6d84cfb59 00007fb6d84cfae9 00007fb6d84cfab1 00007fb6d84cedad
0924 00:22:59.150243 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:59.150288 (+ 45us) write_op.cc:435] Acquired schema lock
0924 00:22:59.150314 (+ 26us) tablet.cc:660] Decoding operations
0924 00:22:59.150788 (+ 474us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:59.150885 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:59.150923 (+ 38us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:59.151161 (+ 238us) tablet.cc:699] Row locks acquired
0924 00:22:59.151186 (+ 25us) write_op.cc:260] PREPARE: finished
0924 00:22:59.151375 (+ 189us) write_op.cc:270] Start()
0924 00:22:59.151652 (+ 277us) write_op.cc:276] Timestamp: P: 1790209379151319 usec, L: 0
0924 00:22:59.151686 (+ 34us) op_driver.cc:331] REPLICATION: starting
0924 00:22:59.158311 (+ 6625us) log.cc:842] Serialized 535 byte log entry
0924 00:22:59.254396 (+ 96085us) op_driver.cc:444] REPLICATION: finished
0924 00:22:59.470075 (+215679us) write_op.cc:301] APPLY: starting
0924 00:22:59.470155 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:59.470245 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:59.470270 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:59.470499 (+ 229us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:59.470683 (+ 184us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:59.470727 (+ 44us) write_op.cc:312] APPLY: finished
0924 00:22:59.470900 (+ 173us) log.cc:842] Serialized 23 byte log entry
0924 00:22:59.471052 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:59.471120 (+ 68us) write_op.cc:454] Released schema lock
0924 00:22:59.471155 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:59.471450 (+ 295us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:59.471482 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1724,"reported_tablets":3,"spinlock_wait_cycles":19840,"child_traces":[["op",{"apply.queue_time_us":215409,"apply.run_cpu_time_us":1636,"apply.run_wall_time_us":1636,"num_ops":1,"prepare.queue_time_us":7357,"prepare.run_cpu_time_us":8720,"prepare.run_wall_time_us":289563,"replication_time_us":102586,"spinlock_wait_cycles":9545600,"thread_start_us":112374,"threads_started":1}]]}
I20260924 00:23:00.470021 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:00.462441 5120 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:00.479223 2145 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 11: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:00.470744 5202 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:23:00.482026 5225 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260924 00:23:00.482123 5266 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.498615 5272 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:00.500245 5141 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:00.515075 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 136759us
I20260924 00:23:00.507053 5202 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:00.526216 2067 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260924 00:23:00.526746 2036 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:00.526747 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.529175 2144 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:00.530330 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:00.534314 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.536592 2137 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:00.534646 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.538905 2139 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:00.538990 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.539582 1989 raft_consensus.cc:3032] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:23:00.539819 1989 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:00.540400 1989 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.542013 1989 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:00.542419 1996 raft_consensus.cc:3032] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:23:00.543015 1996 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:00.390220 5218 rpcz_store.cc:276] 0924 00:22:57.530169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:57.535637 (+ 5468us) service_pool.cc:227] Handling call
0924 00:22:57.843174 (+307537us) ts_tablet_manager.cc:928] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: Initiating tablet copy from peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
0924 00:22:58.083858 (+240684us) spinlock_profiling.cc:243] Waited 6.46 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d1b765fe 00007fb6da9ada39 00007fb6d1b938bb 00007fb6d1b9062a 00007fb6d1b90552 00007fb6d99ad691
0924 00:22:59.524890 (+1441032us) tablet_copy_client.cc:361] Tablet copy session begun
0924 00:23:00.180903 (+656013us) tablet_metadata.cc:641] Metadata flushed
0924 00:23:00.181216 (+ 313us) tablet_copy_client.cc:490] Wrote new tablet metadata
0924 00:23:00.355845 (+174629us) tablet_copy_client.cc:411] Wrote new consensus metadata
0924 00:23:00.356132 (+ 287us) tablet_replica.cc:205] Creating consensus instance
0924 00:23:00.356433 (+ 301us) ts_tablet_manager.cc:983] Replica registered.
0924 00:23:00.356595 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":65351,"spinlock_wait_cycles":14862592,"tablet-copy.queue_time_us":234645,"thread_start_us":454342,"threads_started":2}
I20260924 00:23:00.547518 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.548352 2068 raft_consensus.cc:3037] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:00.527581 5147 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:00.528226 5225 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260924 00:23:00.555043 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.555335 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.555958 1990 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:00.556227 1990 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:00.556512 1994 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260924 00:23:00.556981 1990 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.557415 5202 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.533682 5268 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:23:00.553489 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.569017 1987 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Advancing to term 13
I20260924 00:23:00.554755 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:00.552235 1960 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:00.570005 2138 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:00.535483 5163 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader election won for term 12
I20260924 00:23:00.552666 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:00.565408 2144 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:23:00.545177 1996 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:00.575244 1992 raft_consensus.cc:3032] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:23:00.575444 1996 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:23:00.575584 1992 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:00.571419 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:00.580096 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:00.580670 1989 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:23:00.581533 2110 leader_election.cc:398] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:23:00.581867 2110 leader_election.cc:401] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260924 00:23:00.585834 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:23:00.560043 5322 log.cc:924] Time spent T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260924 00:23:00.576268 1992 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.574405 5301 raft_consensus.cc:2726] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:00.610608 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:00.593318 5223 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:23:00.617396 1987 raft_consensus.cc:2454] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 13.
I20260924 00:23:00.605533 5163 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:00.593451 5145 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:23:00.613336 2138 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:23:00.627866 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:00.628750 2035 leader_election.cc:398] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10.
I20260924 00:23:00.582530 5224 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260924 00:23:00.632470 2110 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:00.632614 2068 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:23:00.629693 2035 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:23:00.614777 5218 tablet_copy_client.cc:791] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 0 data blocks...
I20260924 00:23:00.649463 2109 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:23:00.640878 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:00.607826 5271 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260924 00:23:00.619091 5336 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
W20260924 00:23:00.638305 5332 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260924 00:23:00.655666 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:23:00.653926 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.660876 5147 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:23:00.612394 5272 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260924 00:23:00.649005 5225 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:00.652748 5268 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:23:00.674027 5225 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:00.675902 5218 tablet_copy_client.cc:655] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Starting download of 1 WAL segments...
I20260924 00:23:00.676035 2645 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260924 00:23:00.677448 5225 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:00.679875 5330 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20260924 00:23:00.656942 5141 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260924 00:23:00.668566 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 82593.4us
I20260924 00:23:00.668581 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.683583 2137 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:23:00.690438 2062 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 11: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:00.652719 5313 raft_consensus.cc:2781] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 13 FOLLOWER]: Leader election won for term 13
I20260924 00:23:00.695820 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:00.671808 5305 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:00.700755 2139 raft_consensus.cc:2454] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:23:00.671592 5336 raft_consensus.cc:686] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:00.702059 1960 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:00.657578 5155 consensus_queue.cc:1035] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260924 00:23:00.674250 5147 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:00.714215 5261 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 6. Current term is 7.
I20260924 00:23:00.714425 5268 raft_consensus.cc:686] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:00.750725 5313 raft_consensus.cc:686] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 13 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:00.716814 5297 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260924 00:23:00.684770 5163 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.721474 5340 raft_consensus.cc:2781] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election won for term 7
W20260924 00:23:00.772477 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 60679.8us
W20260924 00:23:00.794535 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 89740.6us
W20260924 00:23:00.582834 5260 consensus_queue.cc:1159] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8c129c01e1c047628f6a87b001aa7258" responder_term: 4 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260924 00:23:00.825078 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:00.840083 5313 consensus_queue.cc:237] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.797189 5297 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:00.805737 5340 raft_consensus.cc:686] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:00.841543 5297 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.842343 5340 consensus_queue.cc:237] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.842473 5336 consensus_queue.cc:237] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.843379 2641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 491) took 1030 ms. Trace:
I20260924 00:23:00.843788 2641 rpcz_store.cc:276] 0924 00:22:59.812771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:59.813495 (+ 724us) service_pool.cc:227] Handling call
0924 00:22:59.813841 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:00.375327 (+561486us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:00.843101 (+467774us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54005ef6d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:00.843287 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1228135168}
I20260924 00:23:00.845266 1996 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260924 00:23:00.845631 1996 pending_rounds.cc:76] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Aborting all ops after (but not including) 5
I20260924 00:23:00.845943 1996 pending_rounds.cc:97] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260924 00:23:00.846210 1996 raft_consensus.cc:2866] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:23:00.828284 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:00.848793 5317 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:00.849453 5186 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:00.851764 5186 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:00.852224 1993 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260924 00:23:00.841953 5335 log.cc:924] Time spent T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
W20260924 00:23:00.841073 5337 log.cc:924] Time spent T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
W20260924 00:23:00.878643 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 49602.8us
I20260924 00:23:00.857889 2142 raft_consensus.cc:1261] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260924 00:23:00.883512 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 76118.8us
I20260924 00:23:00.843648 5297 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:23:00.891719 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 239799us
I20260924 00:23:00.859015 5127 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:00.895960 2067 raft_consensus.cc:1226] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:00.935328 2035 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:23:00.935655 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 45059.5us
I20260924 00:23:00.938275 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:00.938977 2063 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:00.988600 5167 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:23:00.989609 5167 raft_consensus.cc:686] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:00.931670 5127 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.024533 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:01.025189 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:01.038264 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 103027us
I20260924 00:23:01.039705 2035 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:01.045938 5127 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:01.042181 2063 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:23:00.886639 5147 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.089744 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:01.042860 5145 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 00:23:01.095593 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 226742us
I20260924 00:23:00.943172 5282 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
I20260924 00:23:01.100494 5285 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:01.101986 5270 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:01.102677 5270 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:01.101931 5285 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:01.104509 5350 log.cc:924] Time spent T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.162s user 0.000s sys 0.000s
I20260924 00:23:01.104868 5285 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:00.889902 5268 consensus_queue.cc:237] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.103530 5270 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.041484 5167 consensus_queue.cc:237] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:01.132524 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 107435us
I20260924 00:23:01.130787 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:01.022748 5305 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:01.143440 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:01.175770 1960 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:01.181043 5186 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:23:01.181538 5186 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:01.182377 5186 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.186829 5299 raft_consensus.cc:979] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd: : Instructing follower 948356bf66a941ef8249ecbd0ec94f6d to start an election
I20260924 00:23:01.187443 5299 raft_consensus.cc:1067] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Signalling peer 948356bf66a941ef8249ecbd0ec94f6d to start an election
W20260924 00:23:01.205662 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 166032us
I20260924 00:23:01.205689 2063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2730c02ac0a4812841172254e284b90"
dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:47766
I20260924 00:23:01.206629 2035 leader_election.cc:398] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 12. Current term is 13.
I20260924 00:23:01.206616 2063 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:23:01.207165 2063 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
W20260924 00:23:01.228484 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 290768us
W20260924 00:23:01.267722 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 62608us
I20260924 00:23:01.195712 5323 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:01.272447 5323 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.274603 1992 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
W20260924 00:23:01.286374 5314 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.425s user 0.003s sys 0.000s
I20260924 00:23:01.279495 2063 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.287694 5290 consensus_queue.cc:1035] T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:23:01.289220 2063 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:01.290118 5166 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:01.290696 5166 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.301230 2068 raft_consensus.cc:1226] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:01.307272 1992 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 6.
I20260924 00:23:01.310807 5166 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:01.145417 5355 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.177s
I20260924 00:23:01.330307 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:01.337473 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:23:01.360445 5333 log.cc:924] Time spent T 217f6c4b9f5c44d0b3856a8d131557cd P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.838s user 0.002s sys 0.002s
I20260924 00:23:01.366236 1990 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:01.422433 2063 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260924 00:23:01.428767 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 99505us
I20260924 00:23:01.436266 2139 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:23:01.306895 5224 consensus_queue.cc:1035] T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.121s
I20260924 00:23:01.534446 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:01.457806 5318 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:23:01.555229 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 282976us
W20260924 00:23:01.555895 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 91104.8us
I20260924 00:23:01.567905 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:01.574844 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:01.576715 2063 raft_consensus.cc:1261] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260924 00:23:01.609403 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.026s sys 0.062s
W20260924 00:23:01.610082 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.525s user 0.027s sys 0.062s
I20260924 00:23:01.565472 5127 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:01.471935 5266 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260924 00:23:01.655417 1990 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:23:01.446943 5305 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.694365 5323 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:23:01.694116 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 82841.6us
I20260924 00:23:01.617635 5268 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:01.565217 5318 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.658836 5127 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.703824 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:01.804376 5310 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.238s
I20260924 00:23:01.706581 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 2353) took 1147 ms. Trace:
I20260924 00:23:01.815883 1990 rpcz_store.cc:276] 0924 00:23:00.538348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:00.538443 (+ 95us) service_pool.cc:227] Handling call
0924 00:23:01.365581 (+827138us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b58004deab0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4a37f63 00007fb6d4a9baba 00007fb6d4ab33af 00007fb6d4ab4b4e 00007fb6d4ac0228 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c
0924 00:23:01.686067 (+320486us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53601536}
I20260924 00:23:01.817412 2110 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:01.748044 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 51551.1us
W20260924 00:23:01.818318 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 251691us
I20260924 00:23:01.695485 5364 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:01.733165 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:01.763988 5218 tablet_copy_client.cc:531] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:23:01.800359 5283 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:01.829780 5283 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.830487 1995 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260924 00:23:01.831851 5283 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:01.821444 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:01.834041 5383 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260924 00:23:01.835363 5267 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:01.804067 5307 consensus_queue.cc:1035] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.246s
I20260924 00:23:01.836306 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37650 (request call id 202) took 2871 ms. Trace:
I20260924 00:23:01.837568 1893 catalog_manager.cc:5495] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:23:01.838429 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2513) took 1039 ms. Trace:
I20260924 00:23:01.838886 5336 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:01.839514 5336 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.839285 1893 catalog_manager.cc:5495] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:23:01.479915 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260924 00:23:01.733577 5347 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260924 00:23:01.805016 5371 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20260924 00:23:01.829072 5141 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:01.822062 5270 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:01.824391 2110 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:01.840775 1893 catalog_manager.cc:5495] T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260924 00:23:01.838824 1989 rpcz_store.cc:276] 0924 00:23:00.798985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:00.799150 (+ 165us) service_pool.cc:227] Handling call
0924 00:23:00.828215 (+ 29065us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:01.134675 (+306460us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:01.134725 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:01.135316 (+ 591us) log.cc:842] Serialized 26 byte log entry
0924 00:23:01.653579 (+518263us) raft_consensus.cc:1631] Marking committed up to 8
0924 00:23:01.653635 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:01.653741 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:01.838279 (+184538us) raft_consensus.cc:1671] finished
0924 00:23:01.838295 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:01.838385 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38505,"raft.queue_time_us":44024,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":4811,"thread_start_us":736382,"threads_started":2,"wal-append.queue_time_us":582104}
I20260924 00:23:01.796005 5257 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:01.858287 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:01.860507 1893 catalog_manager.cc:5495] T 832ee611b00d4a42b3245beb345af7ad P 8c129c01e1c047628f6a87b001aa7258 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "8c129c01e1c047628f6a87b001aa7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:23:01.853727 5147 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260924 00:23:01.744176 5268 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.861968 5264 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:01.836649 1890 rpcz_store.cc:276] 0924 00:22:58.969035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:58.969257 (+ 222us) service_pool.cc:227] Handling call
0924 00:23:01.836229 (+2866972us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:22:59.306814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:22:59.307337 (+ 523us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:22:59.307372 (+ 35us) write_op.cc:435] Acquired schema lock
0924 00:22:59.307388 (+ 16us) tablet.cc:660] Decoding operations
0924 00:22:59.307715 (+ 327us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:22:59.307760 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0924 00:22:59.307787 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0924 00:22:59.307967 (+ 180us) tablet.cc:699] Row locks acquired
0924 00:22:59.307985 (+ 18us) write_op.cc:260] PREPARE: finished
0924 00:22:59.308127 (+ 142us) write_op.cc:270] Start()
0924 00:22:59.308247 (+ 120us) write_op.cc:276] Timestamp: P: 1790209379308080 usec, L: 0
0924 00:22:59.308270 (+ 23us) op_driver.cc:331] REPLICATION: starting
0924 00:22:59.482265 (+173995us) log.cc:842] Serialized 530 byte log entry
0924 00:22:59.488269 (+ 6004us) op_driver.cc:444] REPLICATION: finished
0924 00:22:59.493399 (+ 5130us) write_op.cc:301] APPLY: starting
0924 00:22:59.493473 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0924 00:22:59.493561 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0924 00:22:59.493656 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:22:59.493866 (+ 210us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:22:59.494003 (+ 137us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 00:22:59.494042 (+ 39us) write_op.cc:312] APPLY: finished
0924 00:22:59.494214 (+ 172us) log.cc:842] Serialized 23 byte log entry
0924 00:22:59.494355 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:59.494427 (+ 72us) write_op.cc:454] Released schema lock
0924 00:22:59.494462 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0924 00:22:59.494729 (+ 267us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:22:59.494757 (+ 28us) write_op.cc:454] Released schema lock
0924 00:23:00.159079 (+664322us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b5400002dd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ab8198 00007fb6d84cbade 00007fb6d84cb2b8 00007fb6d84cfba1 00007fb6d84cfb59 00007fb6d84cfae9 00007fb6d84cfab1 00007fb6d84cedad 00007fb6dadf8204 00007fb6dadf8039
Metrics: {"mutex_wait_us":874778,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":4835,"apply.run_cpu_time_us":1562,"apply.run_wall_time_us":1562,"num_ops":1,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":6704,"prepare.run_wall_time_us":875104,"replication_time_us":179910,"spinlock_wait_cycles":204722048}]]}
I20260924 00:23:01.895912 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:01.896521 1987 raft_consensus.cc:3032] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:23:01.896801 1987 raft_consensus.cc:729] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:01.900043 4606 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:01.897452 1987 consensus_queue.cc:260] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.902180 1987 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:01.842060 5364 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.903074 5299 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:01.913528 5167 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:01.913987 5382 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:01.914371 5374 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:01.868299 5257 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.894273 5159 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260924 00:23:01.915879 5366 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:01.916226 4606 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.862107 5141 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.870368 5270 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.954489 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:23:01.955384 1989 raft_consensus.cc:595] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:01.959865 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 140505us
I20260924 00:23:01.930366 1987 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:23:01.915709 5360 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:01.934743 5318 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:01.977247 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 105441us
I20260924 00:23:01.935684 5299 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:01.981613 4606 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:02.040591 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 56677.5us
I20260924 00:23:02.043895 5394 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:02.046880 2143 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260924 00:23:01.987721 2145 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57748 (request call id 2023) took 2232 ms (client timeout 1970 ms). Trace:
W20260924 00:23:02.048979 2145 rpcz_store.cc:269] 0924 00:22:59.755664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:59.755867 (+ 203us) service_pool.cc:227] Handling call
0924 00:23:01.987482 (+2231615us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b5400350cd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:01.987623 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1539097344}
I20260924 00:23:02.056452 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:02.206497 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 154621us
I20260924 00:23:02.044544 5394 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.218231 5394 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:02.222066 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 243957us
I20260924 00:23:01.939088 5167 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:01.963979 5310 consensus_queue.cc:1035] T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20260924 00:23:02.228760 2068 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260924 00:23:02.057169 5322 log.cc:924] Time spent T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20260924 00:23:02.232079 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 241524us
I20260924 00:23:01.985363 5318 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:01.939666 5382 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.238132 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "8c129c01e1c047628f6a87b001aa7258"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:23:02.255540 1995 raft_consensus.cc:595] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Received request to transfer leadership to TS 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:02.220770 5394 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:02.222078 5186 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:02.200897 5336 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:02.255963 1995 raft_consensus.cc:603] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260924 00:23:02.221823 5155 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:02.284217 2067 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260924 00:23:02.294806 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:02.300457 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 92511.4us
I20260924 00:23:02.301434 2036 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:02.050484 5352 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:02.306670 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:02.302940 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:01.959362 5366 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.310840 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.307505 2062 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:23:01.981356 5360 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.260586 5127 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:02.264273 1960 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.3:36385 timed out after 1.970s (SENT) [suppressed 21 similar messages]
I20260924 00:23:02.316265 1990 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:02.317551 2137 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260924 00:23:02.320427 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:02.322208 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.286098 5317 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:02.324553 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:01.949775 5374 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:02.319386 5375 log.cc:924] Time spent T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.457s user 0.000s sys 0.001s
I20260924 00:23:02.346057 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:02.346369 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:02.299839 5186 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.335929 5257 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.335711 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.336880 5364 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.284888 5394 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.325407 5297 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:02.288458 5318 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.365968 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:02.351635 2137 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:23:02.368069 2137 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:02.364641 5167 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:02.340479 5390 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20260924 00:23:02.307243 5155 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.330735 5268 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:02.342598 5360 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.340969 5401 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:23:02.368779 2137 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.384353 2062 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260924 00:23:02.365317 5396 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:02.375072 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a"
dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
mode: GRACEFUL
new_leader_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
from {username='slave'} at 127.0.0.1:59938
I20260924 00:23:02.384483 2137 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:02.385690 1996 raft_consensus.cc:595] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Received request to transfer leadership to TS 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:02.374956 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:02.381331 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:02.381870 2614 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:02.381897 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.315867 5352 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.332567 5317 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.393358 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.375981 5270 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.374918 5167 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:23:02.394049 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 129816us
I20260924 00:23:02.389474 5396 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.392719 5283 raft_consensus.cc:2726] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:02.408191 5382 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:02.417393 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:23:02.410993 5402 log.cc:924] Time spent T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.364s user 0.000s sys 0.003s
I20260924 00:23:02.372311 5297 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.416566 5167 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.442652 812 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260924 00:23:02.444767 812 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260924 00:23:02.395763 5317 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:02.404500 5141 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.411782 5299 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.457760 2641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 500) took 1278 ms. Trace:
W20260924 00:23:02.442272 5404 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20260924 00:23:02.464983 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 45212us
I20260924 00:23:02.464107 2641 rpcz_store.cc:276] 0924 00:23:01.171883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.172036 (+ 153us) service_pool.cc:227] Handling call
0924 00:23:02.449426 (+1277390us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b5800005a98. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:23:02.449685 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.449916 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.449953 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:02.450010 (+ 57us) raft_consensus.cc:1631] Marking committed up to 8
0924 00:23:02.450055 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:02.450127 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:02.450280 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4330240}
I20260924 00:23:02.415815 5401 raft_consensus.cc:686] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:02.469444 2035 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:02.442139 5318 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:02.472061 2065 raft_consensus.cc:3032] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 LEADER]: Stepping down as leader of term 12
I20260924 00:23:02.442883 5336 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:02.481082 2065 raft_consensus.cc:729] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:02.420995 5266 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260924 00:23:02.504485 2137 raft_consensus.cc:1261] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260924 00:23:02.505113 2137 pending_rounds.cc:76] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Aborting all ops after (but not including) 6
I20260924 00:23:02.505505 2137 pending_rounds.cc:97] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260924 00:23:02.505910 2137 raft_consensus.cc:2866] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:23:02.489670 5163 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:02.518806 5167 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.420460 5358 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260924 00:23:02.478482 5299 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:23:02.478545 5395 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:02.498194 2065 consensus_queue.cc:260] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.543087 5374 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.543264 5342 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:02.527485 5395 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:02.550199 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 64459.6us
I20260924 00:23:02.567250 2035 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:02.524286 5299 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:02.575722 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.526734 5394 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.581449 5352 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:02.493227 5336 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.560629 5374 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:02.615746 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.580024 5299 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:02.574535 5186 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.588476 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.549604 5391 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:02.591292 5267 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:23:02.539885 5155 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:02.594736 5141 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:02.583840 5297 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:02.608697 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.584381 5218 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: Bootstrap starting.
I20260924 00:23:02.616976 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.520386 5401 consensus_queue.cc:237] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.696115 5186 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:02.696892 5186 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.699409 5186 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:02.604189 5396 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:02.721304 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 318375us
W20260924 00:23:02.740787 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 323469us
W20260924 00:23:02.740868 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 413181us
I20260924 00:23:02.746707 2109 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:02.624048 5366 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.549642 5342 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.767540 1989 raft_consensus.cc:1261] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:23:02.761674 2613 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:02.780516 5224 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:02.780501 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:02.721436 5186 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:02.740881 5290 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:02.740410 5363 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:02.754865 5280 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:02.684468 5141 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.761795 5318 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:02.832948 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:02.839471 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 115967us
I20260924 00:23:02.791013 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:02.795775 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.796743 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.630676 5374 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:23:02.852408 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.905514 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.803807 5186 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.636886 5391 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.909699 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:02.921401 2071 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:23:02.921811 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 130908us
I20260924 00:23:02.922789 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.922909 2613 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:02.926499 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 71998.5us
I20260924 00:23:02.825313 5155 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:23:02.938207 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 191607us
I20260924 00:23:02.942812 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.949440 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:02.939635 2109 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:02.951866 5418 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:02.952543 5349 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:02.959859 2071 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
I20260924 00:23:02.838284 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:02.973819 5400 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:02.975354 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:02.982138 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:02.991968 5391 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:02.993239 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:23:03.002470 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 62929.4us
I20260924 00:23:02.928858 5307 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:02.811282 5290 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:02.843026 5318 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:02.927452 5396 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:03.005261 5268 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.039s sys 0.094s
I20260924 00:23:02.945724 5351 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:02.932559 5267 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:02.815128 5363 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.028084 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:03.028276 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:03.028909 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:03.029620 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:23:03.025971 5268 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.040s sys 0.095s
W20260924 00:23:03.024667 5371 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260924 00:23:03.005039 5358 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:23:03.052616 5420 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260924 00:23:03.035274 5380 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.053531 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:03.055428 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.052523 5351 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.107187 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.009344 5297 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:03.011449 5261 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:23:03.113277 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:02.958314 5155 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.125663 5362 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.127170 2136 raft_consensus.cc:2379] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 11: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
W20260924 00:23:03.129040 5424 log.cc:924] Time spent T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.199s user 0.000s sys 0.000s
W20260924 00:23:03.133149 5416 log.cc:924] Time spent T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.374s user 0.001s sys 0.000s
I20260924 00:23:03.049817 5395 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:03.057564 5400 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:03.109365 5421 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:03.147969 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 708) took 1164 ms. Trace:
I20260924 00:23:03.034137 5396 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.130856 5336 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:03.153940 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.164399 1995 raft_consensus.cc:1261] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260924 00:23:03.159641 1987 rpcz_store.cc:276] 0924 00:23:01.983862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.983985 (+ 123us) service_pool.cc:227] Handling call
0924 00:23:01.984264 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.381856 (+397592us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.381888 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:02.381938 (+ 50us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:02.381973 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:03.147637 (+765664us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54002f1a54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:03.147708 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:03.147890 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":913921280}
W20260924 00:23:03.168850 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 210531us
I20260924 00:23:03.147107 5362 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.136613 5297 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.028281 5349 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.118760 5358 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.196527 1991 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:03.197005 5242 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.200966 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.197664 5242 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.207669 5242 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.209861 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:03.210717 5352 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.211233 5352 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:03.005823 5418 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:03.218954 2034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.030s sys 0.098s
W20260924 00:23:03.219764 2034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.729s user 0.030s sys 0.098s
W20260924 00:23:03.186652 5427 log.cc:924] Time spent T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260924 00:23:03.042007 5267 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.226406 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:03.195647 5141 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:03.174635 5421 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:03.238901 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 210890us
I20260924 00:23:03.239354 2066 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:03.217264 5374 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
W20260924 00:23:03.224363 5299 consensus_meta.cc:287] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.587s user 0.005s sys 0.006s
I20260924 00:23:03.132944 5380 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.235219 5421 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:03.197739 5358 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:03.225374 5342 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:03.266062 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.246609 5155 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:03.268224 5362 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.243137 5224 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.268913 5421 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:03.249009 5351 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:03.269989 5421 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.269812 5224 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.274760 5421 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:03.275038 5224 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.244300 5290 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.276295 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.276891 2141 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:23:03.277411 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.276341 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:03.279577 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.250372 5352 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.252245 5363 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.282593 1988 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:23:03.288973 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.265760 5318 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:03.285072 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.285218 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:03.269752 5297 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.299808 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.300601 1987 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:23:03.250973 5299 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.289007 5342 raft_consensus.cc:2726] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:03.305785 2109 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:03.307677 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.308609 5283 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:03.315115 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 2361) took 1423 ms. Trace:
I20260924 00:23:03.315585 5418 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:03.316335 5349 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.339296 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:03.321688 1993 rpcz_store.cc:276] 0924 00:23:01.892121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.894384 (+ 2263us) service_pool.cc:227] Handling call
0924 00:23:01.907203 (+ 12819us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.213488 (+306285us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.213534 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:02.213591 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:23:02.213648 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:03.314885 (+1101237us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b540029a9d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:03.314926 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:03.315059 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":703799168}
W20260924 00:23:03.370664 5393 log.cc:924] Time spent T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.938s user 0.000s sys 0.011s
I20260924 00:23:03.372932 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 705) took 1635 ms. Trace:
I20260924 00:23:03.373363 1994 rpcz_store.cc:276] 0924 00:23:01.737942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.742799 (+ 4857us) service_pool.cc:227] Handling call
0924 00:23:01.815171 (+ 72372us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:01.821716 (+ 6545us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:01.821749 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:01.822252 (+ 503us) log.cc:842] Serialized 26 byte log entry
0924 00:23:02.203613 (+381361us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:02.203670 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:02.203768 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:03.287244 (+1083476us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:23:03.372580 (+ 85336us) raft_consensus.cc:1671] finished
0924 00:23:03.372603 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:03.372866 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6265,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":94,"spinlock_wait_cycles":17968384,"thread_start_us":381006,"threads_started":1,"wal-append.queue_time_us":482310}
I20260924 00:23:03.377323 5349 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.378091 5349 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.378875 5137 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.379459 5137 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.379914 5380 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.380481 5349 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:03.330895 5283 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:03.385552 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:03.386304 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:03.328552 5313 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260924 00:23:03.329164 5401 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.012s sys 0.028s
I20260924 00:23:03.384929 5283 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:03.423148 5300 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
W20260924 00:23:03.440774 5401 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s user 0.013s sys 0.028s
I20260924 00:23:03.380517 5380 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:03.424170 5313 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:03.562077 2071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.013s sys 0.041s
W20260924 00:23:03.562582 2071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.013s sys 0.041s
I20260924 00:23:03.449649 2643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45594 (request call id 628) took 1834 ms. Trace:
I20260924 00:23:03.469588 5240 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:03.560992 5313 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:23:03.465404 5300 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:03.920087 5402 log.cc:924] Time spent T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.382s user 0.000s sys 0.000s
I20260924 00:23:04.078545 5362 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:23:04.079077 5433 log.cc:924] Time spent T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20260924 00:23:04.105358 5362 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:03.524842 5380 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.095306 5154 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:04.116396 5154 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.100447 5299 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.102695 5388 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:04.057839 5145 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:04.032546 5143 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:03.960696 5318 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:04.192229 5154 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.167593 5145 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.121608 5388 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.180037 5143 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.201913 5318 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:04.252107 1990 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.100s user 0.014s sys 0.060s
I20260924 00:23:04.317538 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:04.327379 5300 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:04.328351 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43194 (request call id 752) took 1049 ms. Trace:
I20260924 00:23:04.310974 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.310204 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.329847 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.330201 2145 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:23:04.330652 5145 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.331176 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.358639 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.360570 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.309275 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:04.362926 2644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45594 (request call id 642) took 1043 ms. Trace:
I20260924 00:23:04.363508 2644 rpcz_store.cc:276] 0924 00:23:03.319210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:03.320645 (+ 1435us) service_pool.cc:227] Handling call
0924 00:23:04.362766 (+1042121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:04.310577 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.365547 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.366389 2137 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:23:04.367357 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.368995 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.332966 5388 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.335557 5143 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:04.250110 2645 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.015s sys 0.003s
W20260924 00:23:04.373545 2645 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s user 0.016s sys 0.004s
I20260924 00:23:04.374401 5408 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260924 00:23:04.327692 1990 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.176s user 0.015s sys 0.061s
W20260924 00:23:04.378875 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.232s user 0.020s sys 0.053s
W20260924 00:23:04.379982 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.238s user 0.020s sys 0.054s
I20260924 00:23:04.339885 5318 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.341665 2069 rpcz_store.cc:276] 0924 00:23:03.278663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:03.386587 (+107924us) service_pool.cc:227] Handling call
0924 00:23:04.328278 (+941691us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:04.382443 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1922) took 1508 ms. Trace:
I20260924 00:23:04.382833 2069 rpcz_store.cc:276] 0924 00:23:02.873816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.381461 (+1507645us) service_pool.cc:227] Handling call
0924 00:23:04.381834 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:04.382050 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.382084 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.382134 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:23:04.382182 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.382246 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.382386 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:04.348228 5154 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:04.385002 5154 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.387209 5154 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:04.355669 1960 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.054s user 0.017s sys 0.051s
W20260924 00:23:04.390343 1960 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.106s user 0.018s sys 0.051s
W20260924 00:23:04.390452 5377 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.077s user 0.023s sys 0.047s
W20260924 00:23:04.391696 5377 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.094s user 0.024s sys 0.047s
I20260924 00:23:04.392737 5143 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:04.393517 5143 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.396216 5143 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:04.390772 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 1.10686e+06us
W20260924 00:23:04.397120 5442 log.cc:924] Time spent T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
W20260924 00:23:04.354038 1959 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.060s user 0.017s sys 0.037s
W20260924 00:23:04.400369 1959 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.107s user 0.017s sys 0.037s
I20260924 00:23:04.396857 5446 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:04.400933 2136 raft_consensus.cc:1261] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260924 00:23:04.389564 2144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 2055) took 2071 ms. Trace:
I20260924 00:23:04.402144 1991 raft_consensus.cc:1226] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:04.405012 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.370985 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.361754 2142 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 11.
W20260924 00:23:03.939733 5207 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.032s sys 0.028s
I20260924 00:23:04.423307 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.413117 2144 rpcz_store.cc:276] 0924 00:23:02.317944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.318068 (+ 124us) service_pool.cc:227] Handling call
0924 00:23:02.318359 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:02.607643 (+289284us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.607673 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:02.608075 (+ 402us) log.cc:842] Serialized 26 byte log entry
0924 00:23:03.031149 (+423074us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:23:03.031182 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:03.031241 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:04.106523 (+1075282us) raft_consensus.cc:1671] finished
0924 00:23:04.106549 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.389474 (+282925us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":463256,"raft.queue_time_us":289194,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":15436,"thread_start_us":407643,"threads_started":1,"wal-append.queue_time_us":526895}
I20260924 00:23:04.440759 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.441871 1993 raft_consensus.cc:2379] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 7: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:04.443169 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:23:04.428339 5207 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.654s user 0.032s sys 0.029s
I20260924 00:23:04.522580 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:03.921962 5240 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.440063 5352 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:04.546847 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 147124us
I20260924 00:23:04.541224 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2540) took 1558 ms. Trace:
I20260924 00:23:04.548707 1990 rpcz_store.cc:276] 0924 00:23:02.982719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:03.119409 (+136690us) service_pool.cc:227] Handling call
0924 00:23:03.137038 (+ 17629us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:03.137224 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 9
0924 00:23:04.378709 (+1241485us) log.cc:842] Serialized 14 byte log entry
0924 00:23:04.540825 (+162116us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.540874 (+ 49us) raft_consensus.cc:1631] Marking committed up to 9
0924 00:23:04.540932 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.541011 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.541159 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1196227,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":1388382,"threads_started":2}
I20260924 00:23:04.549927 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2523) took 2964 ms. Trace:
I20260924 00:23:04.578011 1960 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:04.447059 5318 consensus_queue.cc:1035] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.097s
I20260924 00:23:04.581676 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.577962 1992 rpcz_store.cc:276] 0924 00:23:01.579971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.600630 (+ 20659us) service_pool.cc:227] Handling call
0924 00:23:01.601032 (+ 402us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:01.601468 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:01.601522 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:03.029868 (+1428346us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b2c000287c8. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d349fe11 00007fb6d34af442 00007fb6d4af7c11 00007fb6d4abc58a 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772
0924 00:23:03.042143 (+ 12275us) log.cc:842] Serialized 26 byte log entry
0924 00:23:03.465810 (+423667us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:23:03.465849 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:03.465918 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:04.544147 (+1078229us) raft_consensus.cc:1671] finished
0924 00:23:04.544169 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.544669 (+ 500us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56586,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":8952,"spinlock_wait_cycles":4176640,"thread_start_us":338609,"threads_started":1,"wal-append.queue_time_us":1261656}
I20260924 00:23:04.582548 1993 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:04.596580 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.522013 5143 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:04.599603 5143 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:04.618301 2614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.029s sys 0.022s
W20260924 00:23:04.618947 2614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.540s user 0.029s sys 0.023s
I20260924 00:23:04.630986 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.631924 1994 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:04.421557 5446 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.543258 5145 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:04.597544 2144 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:23:04.657503 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1924) took 1732 ms. Trace:
I20260924 00:23:04.661572 2070 rpcz_store.cc:276] 0924 00:23:02.925561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.656340 (+1730779us) service_pool.cc:227] Handling call
0924 00:23:04.656812 (+ 472us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:04.657046 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.657083 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.657158 (+ 75us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:04.657209 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.657283 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.657423 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260924 00:23:04.660146 2612 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.009s sys 0.049s
I20260924 00:23:04.575346 5366 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:04.669159 2140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 596) took 2297 ms. Trace:
W20260924 00:23:04.663681 5441 log.cc:924] Time spent T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.354s user 0.000s sys 0.001s
I20260924 00:23:04.670471 2140 rpcz_store.cc:276] 0924 00:23:02.368815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.369338 (+ 523us) service_pool.cc:227] Handling call
0924 00:23:02.373942 (+ 4604us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.391305 (+ 17363us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.391450 (+ 145us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:02.391523 (+ 73us) raft_consensus.cc:1631] Marking committed up to 8
0924 00:23:02.391577 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.665977 (+2274400us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b5400012cd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:04.666031 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.666156 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39452032}
I20260924 00:23:04.579164 5372 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:03.790056 2643 rpcz_store.cc:276] 0924 00:23:01.615242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.616765 (+ 1523us) service_pool.cc:227] Handling call
0924 00:23:01.743477 (+126712us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:01.953898 (+210421us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:01.958712 (+ 4814us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:01.974869 (+ 16157us) raft_consensus.cc:1631] Marking committed up to 7
0924 00:23:01.979193 (+ 4324us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:03.449349 (+1470156us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b540050da54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:03.449405 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:03.449582 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":450771200}
I20260924 00:23:04.634791 5447 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:23:04.672071 2141 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57748 (request call id 2063) took 1640 ms (client timeout 2184 ms). Trace:
W20260924 00:23:04.706125 2141 rpcz_store.cc:269] 0924 00:23:03.028502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:03.028651 (+ 149us) service_pool.cc:227] Handling call
0924 00:23:04.668149 (+1639498us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b5400330ed4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:04.668254 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":559698816}
W20260924 00:23:04.666596 2612 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.588s user 0.009s sys 0.049s
I20260924 00:23:04.547878 5352 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.738425 5154 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:04.739100 5154 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.739842 5352 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:04.741884 5154 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.730839 5372 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.746779 5372 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.748019 5372 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:04.748453 5372 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:04.736398 5450 log.cc:924] Time spent T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260924 00:23:04.665639 5145 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.737725 5306 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:04.750622 2645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45582 (request call id 651) took 2883 ms. Trace:
I20260924 00:23:04.671547 5366 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.657598 5388 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:04.750722 5306 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.751190 5282 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:04.871003 5366 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.869586 2645 rpcz_store.cc:276] 0924 00:23:01.866920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.869293 (+ 2373us) service_pool.cc:227] Handling call
0924 00:23:01.894740 (+ 25447us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:01.951258 (+ 56518us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:01.953811 (+ 2553us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:03.369957 (+1416146us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b2c00127978. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d349fe11 00007fb6d34af442 00007fb6d4af7c11 00007fb6d4abc58a 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772
0924 00:23:03.370351 (+ 394us) log.cc:842] Serialized 26 byte log entry
0924 00:23:04.374096 (+1003745us) raft_consensus.cc:1631] Marking committed up to 7
0924 00:23:04.374160 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.389789 (+ 15629us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:04.750193 (+360404us) raft_consensus.cc:1671] finished
0924 00:23:04.750238 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.750519 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18862,"raft.queue_time_us":22211,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":15427,"spinlock_wait_cycles":27146112,"thread_start_us":1003261,"threads_started":1,"wal-append.queue_time_us":1074553}
I20260924 00:23:04.871166 5282 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.769595 5434 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:04.872260 5306 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.831389 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.873268 5282 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.872452 5434 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.877759 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.900416 2141 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:23:04.752039 5145 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:04.826085 5459 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:04.841038 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.901841 5434 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:04.902402 5459 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.905092 5459 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:04.905575 5446 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:04.916319 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.916636 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.916952 1992 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:04.917224 1993 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:23:04.777014 5372 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.886881 5388 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.923559 5388 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.822358 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:04.925274 2642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45582 (request call id 657) took 2135 ms (client timeout 1702 ms). Trace:
I20260924 00:23:04.927054 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.927919 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.928651 2144 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:04.911494 2613 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:04.930830 5282 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:23:04.930924 2109 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:04.931258 5282 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:04.931689 5282 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:04.932978 5351 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:04.836695 5313 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.911674 5458 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:04.936544 5313 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:04.909466 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:04.939079 2140 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 9.
I20260924 00:23:04.910368 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 2286) took 2548 ms. Trace:
I20260924 00:23:04.940436 5282 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.942133 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 2064) took 1913 ms. Trace:
I20260924 00:23:04.943147 5282 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.942696 2139 rpcz_store.cc:276] 0924 00:23:03.028851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:03.028967 (+ 116us) service_pool.cc:227] Handling call
0924 00:23:03.053029 (+ 24062us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:03.157977 (+104948us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:03.158023 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:03.158081 (+ 58us) raft_consensus.cc:1631] Marking committed up to 8
0924 00:23:03.158124 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.941901 (+1783777us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b5400371254. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:04.941947 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.942068 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":240733568}
I20260924 00:23:04.940189 2068 rpcz_store.cc:276] 0924 00:23:02.361690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.361808 (+ 118us) service_pool.cc:227] Handling call
0924 00:23:02.377736 (+ 15928us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.415776 (+ 38040us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.415806 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:02.415895 (+ 89us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:23:02.415939 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.910123 (+2494184us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b5400347954. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:04.910171 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.910295 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87029120}
I20260924 00:23:04.946720 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 749) took 1740 ms. Trace:
I20260924 00:23:04.946987 2068 rpcz_store.cc:276] 0924 00:23:03.206661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.946040 (+1739379us) service_pool.cc:227] Handling call
0924 00:23:04.946279 (+ 239us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:04.946461 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.946486 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.946524 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:04.946561 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.946605 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.946681 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:04.948113 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 753) took 1666 ms. Trace:
I20260924 00:23:04.948212 2641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 507) took 2204 ms. Trace:
I20260924 00:23:04.948374 2068 rpcz_store.cc:276] 0924 00:23:03.281430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.947579 (+1666149us) service_pool.cc:227] Handling call
0924 00:23:04.947782 (+ 203us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:04.947900 (+ 118us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.947923 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.947958 (+ 35us) raft_consensus.cc:1631] Marking committed up to 9
0924 00:23:04.947985 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.948015 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.948077 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:04.948642 2641 rpcz_store.cc:276] 0924 00:23:02.744006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.744502 (+ 496us) service_pool.cc:227] Handling call
0924 00:23:02.744829 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:03.047751 (+302922us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:03.047786 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:03.047822 (+ 36us) raft_consensus.cc:1631] Marking committed up to 8
0924 00:23:03.047855 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.948003 (+1900148us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b540071e8d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:04.948046 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.948158 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":696165120}
I20260924 00:23:04.950503 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 754) took 1630 ms. Trace:
I20260924 00:23:04.949963 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 2369) took 2181 ms. Trace:
W20260924 00:23:04.951026 5405 log.cc:924] Time spent T ed845c50caec4187b2daed75db23bfd8 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 2.615s user 0.013s sys 0.021s
W20260924 00:23:04.925876 2642 rpcz_store.cc:269] 0924 00:23:02.789456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.790012 (+ 556us) service_pool.cc:227] Handling call
0924 00:23:04.822137 (+2032125us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b5800005a98. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:23:04.925177 (+103040us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30182656}
I20260924 00:23:04.950949 2068 rpcz_store.cc:276] 0924 00:23:03.320585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.949783 (+1629198us) service_pool.cc:227] Handling call
0924 00:23:04.950069 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:04.950217 (+ 148us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.950246 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.950280 (+ 34us) raft_consensus.cc:1631] Marking committed up to 9
0924 00:23:04.950313 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.950365 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.950457 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:04.957419 5267 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:04.892162 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1925) took 1895 ms. Trace:
I20260924 00:23:04.958184 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 2293) took 2197 ms. Trace:
I20260924 00:23:04.912549 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1921) took 2060 ms. Trace:
I20260924 00:23:04.953577 2143 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57748 (request call id 2048) took 3523 ms. Trace:
I20260924 00:23:04.959372 2143 rpcz_store.cc:276] 0924 00:23:01.430473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:01.433081 (+ 2608us) service_pool.cc:227] Handling call
0924 00:23:02.046497 (+613416us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.048174 (+ 1677us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.953413 (+2905239us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b5400350cd0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abba09 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:04.953530 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1228484992}
I20260924 00:23:04.961215 5143 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:04.960745 2071 rpcz_store.cc:276] 0924 00:23:02.851695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:03.015367 (+163672us) service_pool.cc:227] Handling call
0924 00:23:03.015744 (+ 377us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:03.563077 (+547333us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:03.563115 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:04.326672 (+763557us) log.cc:842] Serialized 26 byte log entry
0924 00:23:04.667086 (+340414us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:23:04.667135 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.667225 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:04.912284 (+245059us) raft_consensus.cc:1671] finished
0924 00:23:04.912309 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.912464 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114497,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9538,"thread_start_us":886685,"threads_started":2,"wal-append.queue_time_us":503508}
W20260924 00:23:04.955483 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 379666us
I20260924 00:23:04.977069 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 748) took 2277 ms. Trace:
I20260924 00:23:04.977447 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:04.977643 2064 rpcz_store.cc:276] 0924 00:23:02.700107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.801608 (+101501us) service_pool.cc:227] Handling call
0924 00:23:02.802004 (+ 396us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:03.238795 (+436791us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:03.238823 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:03.238863 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:03.238903 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.976798 (+1737895us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b5400348854. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:04.976855 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.977003 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1004091264}
I20260924 00:23:04.978508 5342 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:04.979290 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:04.980706 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:04.912815 5362 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:04.981868 2064 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 9.
I20260924 00:23:04.982775 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:04.983371 2064 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
W20260924 00:23:04.983800 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 103706us
I20260924 00:23:04.984171 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:04.955654 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:04.985117 2110 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:04.985594 2140 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:04.936408 5458 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:04.986914 5362 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:04.987314 5362 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:04.987686 5362 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:04.954783 2613 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:04.962626 5261 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:04.958515 2070 rpcz_store.cc:276] 0924 00:23:02.996813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.669297 (+1672484us) service_pool.cc:227] Handling call
0924 00:23:04.891570 (+222273us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:04.891758 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.891798 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.891852 (+ 54us) raft_consensus.cc:1631] Marking committed up to 8
0924 00:23:04.891895 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.891959 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.892093 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:04.968207 5240 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:04.970498 5360 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:04.971310 5399 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.631s user 0.014s sys 0.034s
I20260924 00:23:04.971289 5270 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:04.972277 5432 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:04.974181 5412 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:04.958722 2066 rpcz_store.cc:276] 0924 00:23:02.761055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.834390 (+ 73335us) service_pool.cc:227] Handling call
0924 00:23:02.834646 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:03.245694 (+411048us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.940470 (+1694776us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400155ed4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:04.958105 (+ 17635us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921702272}
I20260924 00:23:04.951277 1989 rpcz_store.cc:276] 0924 00:23:02.768879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.769001 (+ 122us) service_pool.cc:227] Handling call
0924 00:23:02.769354 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:03.110389 (+341035us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:03.110431 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:03.110484 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:23:03.110521 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.949772 (+1839251us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54000744d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:04.949813 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:04.949914 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":783866752}
I20260924 00:23:04.958798 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:47762 (request call id 1051) took 1737 ms. Trace:
I20260924 00:23:05.028223 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:04.995743 5380 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:04.996127 5261 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:04.996523 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.007951 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.009112 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.034869 1991 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:23:05.011655 5346 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:05.012481 5387 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:23:05.014701 5399 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.663s user 0.014s sys 0.035s
I20260924 00:23:05.002929 5266 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:04.989256 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 58436.9us
I20260924 00:23:05.066018 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.066251 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.067219 2109 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.067322 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:23:05.030144 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 49763.1us
I20260924 00:23:05.028869 2068 rpcz_store.cc:276] 0924 00:23:03.221097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.957667 (+1736570us) service_pool.cc:227] Handling call
0924 00:23:04.958743 (+ 1076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:05.031731 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:04.988577 5458 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.037039 5464 consensus_queue.cc:1046] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260924 00:23:05.038699 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.036405 2642 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:23:05.012362 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:23:05.029032 2035 connection.cc:232] RPC call timeout handler was delayed by -1.0092373539998789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
I20260924 00:23:05.074237 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 2280) took 2760 ms. Trace:
I20260924 00:23:05.074249 1996 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:23:04.973012 5186 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.058897 5461 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:05.086732 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.060010 5261 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.059890 2110 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.060796 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.085747 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:05.071921 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 89502.3us
I20260924 00:23:05.071954 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.072752 2613 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:05.074827 2062 rpcz_store.cc:276] 0924 00:23:02.313663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.313948 (+ 285us) service_pool.cc:227] Handling call
0924 00:23:02.314315 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.384915 (+ 70600us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:05.073873 (+2688958us) spinlock_profiling.cc:243] Waited 69.7 ms on lock 0x7b54002b2d54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:05.074161 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":160268672}
I20260924 00:23:05.079195 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.099960 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.100543 2642 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:05.100622 2613 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:05.080511 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.030920 5143 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:05.017652 5342 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:05.028247 5271 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:05.068007 1992 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 9.
I20260924 00:23:05.034297 5401 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:05.082549 5351 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:05.101635 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:05.103350 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.014219 5360 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:05.106053 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 46288.4us
I20260924 00:23:05.106112 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.107242 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.104954 5464 raft_consensus.cc:3032] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:23:05.065522 5346 raft_consensus.cc:482] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.102114 2641 raft_consensus.cc:2379] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:23:05.124414 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.032583 5270 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.096673 5266 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.087122 5311 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:23:05.134018 2137 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [term 13 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 12. Current term is 13. Ops: []
I20260924 00:23:05.111109 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.114152 1996 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:23:05.115155 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.121095 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.112203 5342 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.136940 2137 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.125864 2110 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.086244 5387 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.127950 5462 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:05.129074 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 2285) took 2775 ms. Trace:
I20260924 00:23:05.129844 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.122797 5306 raft_consensus.cc:2726] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:05.116619 5464 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:05.030740 5432 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.109833 5307 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:05.133381 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.144017 5307 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.133473 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.144423 5307 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.113430 5313 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:05.110113 2035 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.109617 5143 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.104552 5461 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.140126 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.113189 5271 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.142033 2139 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260924 00:23:05.113814 5401 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.133424 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.143173 2136 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:23:05.145823 2109 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:23:05.129472 5377 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:05.137915 2110 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.167987 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.150074 2614 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:05.150990 5267 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:05.151872 5360 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.154237 5464 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.161931 2140 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:23:05.142889 2067 rpcz_store.cc:276] 0924 00:23:02.353896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.354156 (+ 260us) service_pool.cc:227] Handling call
0924 00:23:02.354415 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.361265 (+ 6850us) raft_consensus.cc:1520] Early marking committed up to index 9
0924 00:23:02.471728 (+110463us) log.cc:842] Serialized 14 byte log entry
0924 00:23:02.754574 (+282846us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:02.754641 (+ 67us) raft_consensus.cc:1631] Marking committed up to 9
0924 00:23:02.754688 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:05.128862 (+2374174us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b54000b2054. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:05.128905 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:05.129024 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61459,"prepare.queue_time_us":109419,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":40908,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":14133,"spinlock_wait_cycles":15054208,"thread_start_us":345964,"threads_started":2,"wal-append.queue_time_us":674986,"wal-append.run_cpu_time_us":1506,"wal-append.run_wall_time_us":1009407}
I20260924 00:23:05.163410 2070 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:23:05.125609 5346 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:05.172572 5352 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:23:05.149369 5389 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:05.141688 5147 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:05.169092 2035 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.167883 2144 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 12.
I20260924 00:23:05.049672 5412 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.173126 1995 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:23:05.173705 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 594) took 2814 ms. Trace:
I20260924 00:23:05.174160 2614 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:05.172600 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.174515 2613 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:05.148089 5313 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.181226 2614 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:05.182137 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.183039 2109 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:05.175849 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:05.171371 5377 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.184674 5271 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.184939 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.175573 5122 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:05.185710 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.186012 5122 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.186091 5377 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.186450 5122 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.186661 2070 raft_consensus.cc:2396] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260924 00:23:05.188341 5290 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:05.180441 2138 rpcz_store.cc:276] 0924 00:23:02.358902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.359140 (+ 238us) service_pool.cc:227] Handling call
0924 00:23:02.359465 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:02.468744 (+109279us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.468788 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:02.468842 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:02.468902 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:05.173053 (+2704151us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540037fad4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:05.173100 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:05.173657 (+ 557us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":250767232}
I20260924 00:23:05.191030 2614 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:05.194551 5471 log.cc:924] Time spent T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260924 00:23:05.170352 5394 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:05.161379 5462 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.179577 5389 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:05.171015 5458 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:05.170199 5267 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.196166 2035 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.198380 5266 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.173022 5352 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.189108 5382 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:23:05.206135 2109 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:05.179915 5147 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.189455 5313 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.183246 5360 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:05.218995 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.190359 5270 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.222119 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:05.200024 5290 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:05.224887 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.201020 5377 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:05.227792 1994 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 5.
I20260924 00:23:05.228108 2109 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.225811 2614 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:05.224108 5412 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.197963 5464 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:05.203266 5458 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.230273 2614 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:05.202988 5389 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:05.203174 5267 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.208571 5120 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260924 00:23:05.233901 5411 log.cc:924] Time spent T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 2.722s user 0.006s sys 0.023s
I20260924 00:23:05.211124 5352 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:05.235118 5151 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:23:05.235618 5207 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:05.236717 2035 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.220203 5306 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:05.245405 5351 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:05.210471 5261 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260924 00:23:05.220923 5432 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.224721 5147 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.226967 5311 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:05.224974 5360 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.227067 5460 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260924 00:23:05.229573 5290 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:05.245872 5462 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.242740 5120 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.230818 5382 raft_consensus.cc:686] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 5 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:05.244356 5458 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.219352 5394 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260924 00:23:05.252425 5351 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:05.250253 5151 raft_consensus.cc:482] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.266872 2035 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.252844 5306 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.266098 5120 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:05.272131 5306 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.272327 5461 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.259732 5387 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.263192 5360 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:05.263151 5270 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:05.276297 5143 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.278024 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1910) took 3054 ms. Trace:
I20260924 00:23:05.264940 5290 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:05.273308 5151 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:05.272931 5351 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:05.275100 5342 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.287833 2063 rpcz_store.cc:276] 0924 00:23:02.222895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.223220 (+ 325us) service_pool.cc:227] Handling call
0924 00:23:02.223678 (+ 458us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:02.247388 (+ 23710us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:02.247440 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:02.282831 (+ 35391us) log.cc:842] Serialized 26 byte log entry
0924 00:23:02.347459 (+ 64628us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:02.347515 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:02.347610 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:05.131063 (+2783453us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:23:05.275048 (+143985us) raft_consensus.cc:1671] finished
0924 00:23:05.275062 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:05.277616 (+ 2554us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":31230,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":18402,"spinlock_wait_cycles":6775680,"thread_start_us":24358,"threads_started":1,"wal-append.queue_time_us":85672}
I20260924 00:23:05.253468 5122 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.299405 2035 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.289583 5461 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:05.287180 5270 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.284833 5401 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.302896 2035 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.303210 5462 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:23:05.305110 5462 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.305557 5462 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:05.310371 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.310820 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.311091 1991 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:05.311465 1987 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:05.311910 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.312244 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.312615 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.313273 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.313733 1988 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260924 00:23:05.314791 2110 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.317561 2065 raft_consensus.cc:3037] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Advancing to term 13
I20260924 00:23:05.322429 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.323448 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.324350 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.302649 5270 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:05.325994 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.325945 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.328292 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.328982 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.329844 2136 raft_consensus.cc:2454] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:23:05.302518 5461 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.332139 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.332780 1994 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:05.334208 2110 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.335870 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:05.338399 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.335881 5461 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:05.339926 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.340186 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.309623 5401 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260924 00:23:05.331403 5396 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.341872 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.348268 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.342739 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.344877 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.345749 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.302815 5313 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.342155 2035 leader_election.cc:302] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.350898 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.354384 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.354900 5122 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.355296 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.299790 5445 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260924 00:23:05.348455 5401 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.366907 5401 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Advancing to term 13
I20260924 00:23:05.340303 5261 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:05.346539 5460 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:05.361694 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:05.362726 5479 log.cc:924] Time spent T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260924 00:23:05.359349 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.369732 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.361591 5362 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.373358 2110 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.374895 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.376631 5366 raft_consensus.cc:2726] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:05.370301 5261 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.380254 5307 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.383769 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.357034 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.389366 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.350996 5137 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.404556 1994 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.375701 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:05.373449 5460 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:05.422896 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.339640 5382 consensus_queue.cc:237] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.415616 5218 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:05.416990 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.407644 5380 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:23:05.414348 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.427739 2065 raft_consensus.cc:1261] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260924 00:23:05.405642 5261 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.374357 5297 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260924 00:23:05.434428 2139 raft_consensus.cc:2379] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 3: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:05.425061 5430 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:05.420781 2140 raft_consensus.cc:1226] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260924 00:23:05.436982 2065 pending_rounds.cc:76] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 8
I20260924 00:23:05.437433 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.455294 2065 pending_rounds.cc:97] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260924 00:23:05.455634 2065 raft_consensus.cc:2866] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:23:05.437916 1994 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:23:05.441318 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.456868 2067 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
I20260924 00:23:05.435110 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.457259 2035 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.424160 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.426061 5460 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.477028 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.473518 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.473941 5432 consensus_queue.cc:1046] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:23:05.474151 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.460759 2614 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:23:05.478518 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 48400.5us
I20260924 00:23:05.479180 2110 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.410390 5346 raft_consensus.cc:504] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:05.480288 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 44016.7us
I20260924 00:23:05.481611 2035 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.443840 5167 raft_consensus.cc:2781] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:05.436669 5310 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.483084 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.487428 2136 raft_consensus.cc:2379] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:05.487607 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:05.456167 5369 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:05.472124 5396 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:23:05.490872 5418 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:05.472380 5366 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:05.491645 5310 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:05.492452 5387 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:23:05.493427 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.493505 5352 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.494061 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.494416 5313 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.484450 5407 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:23:05.502367 5307 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.506361 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:05.508898 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:05.489934 2035 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.470078 5430 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.553937 1991 raft_consensus.cc:1226] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 12. Current term is 13. Ops: []
I20260924 00:23:05.489737 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.511907 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.515496 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:05.518731 2641 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:05.530674 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.537559 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.500849 5306 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.524549 5387 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:05.545334 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.560206 5387 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:05.560425 5485 raft_consensus.cc:482] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:05.555174 5432 raft_consensus.cc:2726] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:05.562507 5267 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.549582 2139 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260924 00:23:05.564761 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.517972 5380 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260924 00:23:05.551483 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 61613.1us
I20260924 00:23:05.552274 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.552646 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.487480 5167 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:05.555217 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.572378 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.574606 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.558578 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.575587 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.576897 1989 raft_consensus.cc:2379] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:05.559317 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.575441 2143 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.568408 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.485487 5290 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.558578 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.576714 2064 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:23:05.569821 5218 tablet_bootstrap.cc:492] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: Bootstrap complete.
I20260924 00:23:05.583129 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.555855 5430 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
W20260924 00:23:05.585682 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 100306us
I20260924 00:23:05.589180 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.586056 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.517470 5369 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.585728 1995 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:05.586948 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.556412 5407 raft_consensus.cc:686] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:05.571537 5267 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.592934 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.592368 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.588482 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.579144 5362 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.579440 5167 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.571072 5380 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:05.560956 5485 raft_consensus.cc:504] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.591709 5483 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:23:05.611452 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.614152 2143 raft_consensus.cc:2454] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:23:05.621884 2648 raft_consensus.cc:1226] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [term 5 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:05.622884 5346 leader_election.cc:288] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.615491 5360 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.627179 2035 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.569672 5461 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.623173 5458 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.629977 2142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 2402) took 1284 ms. Trace:
I20260924 00:23:05.676168 2143 raft_consensus.cc:1226] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:05.690769 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.683923 2142 rpcz_store.cc:276] 0924 00:23:04.345304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.420754 (+ 75450us) service_pool.cc:227] Handling call
0924 00:23:04.668387 (+247633us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:04.668544 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.668565 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:04.668601 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0924 00:23:04.668627 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:04.668673 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:05.629829 (+961156us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400330ed0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abba09 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:05.629929 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":568647040}
I20260924 00:23:05.628461 5218 ts_tablet_manager.cc:1397] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: Time spent bootstrapping tablet: real 3.044s user 0.035s sys 0.023s
I20260924 00:23:05.623750 5380 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.694299 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.628093 5483 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:05.731902 5483 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:05.617995 5147 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.712116 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c6ea1617904bf9bf2b2d904d0e2ae5" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.596514 5389 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.628736 5418 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:23:05.722568 2070 raft_consensus.cc:1226] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:05.696951 5394 raft_consensus.cc:2781] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:23:05.760125 5394 raft_consensus.cc:686] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:05.761088 5394 consensus_queue.cc:237] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.762527 5389 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.744802 2143 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:05.766350 2139 raft_consensus.cc:1226] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:05.747437 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.773283 2143 raft_consensus.cc:2454] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 10.
I20260924 00:23:05.777675 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.744027 5462 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.783092 5401 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.800881 2141 raft_consensus.cc:1226] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:05.717048 5493 consensus_queue.cc:1046] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:23:05.785267 2642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:05.807300 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.784323 5458 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.803604 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.808919 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.746308 5310 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.794417 5485 leader_election.cc:288] T ba683df200644d888896fb84dbc748e8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.824684 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.763661 5460 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.830147 2138 raft_consensus.cc:1226] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260924 00:23:05.830865 5462 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.830523 5218 raft_consensus.cc:348] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:05.831449 5218 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Initialized, Role: LEARNER
I20260924 00:23:05.832523 5401 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 13 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.832082 5218 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:05.793499 5483 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.834034 5460 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.785979 5352 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.803042 2644 raft_consensus.cc:1226] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:05.838395 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.795579 5461 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.840838 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.842540 1987 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:05.842515 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.836051 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:05.848063 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:23:05.845348 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 247741us
I20260924 00:23:05.850670 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.859967 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.860823 1993 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:05.862387 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:05.870273 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.871021 1989 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.871037 1987 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:23:05.839198 5483 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.874738 2614 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:05.876457 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.877068 1995 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:05.878104 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.841748 5306 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.879335 1996 raft_consensus.cc:1226] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:05.847601 5290 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.890075 2140 raft_consensus.cc:1226] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260924 00:23:05.786249 5407 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:05.894692 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 75132.1us
I20260924 00:23:05.900290 2613 leader_election.cc:302] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:23:05.894284 1989 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:05.903342 1995 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:05.912050 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:05.890480 5282 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:23:05.858793 5430 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.914011 2110 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:05.923040 2648 raft_consensus.cc:1226] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:05.925809 5224 raft_consensus.cc:2781] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:05.926332 5224 raft_consensus.cc:686] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:05.926708 5218 ts_tablet_manager.cc:1428] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf: Time spent starting tablet: real 0.195s user 0.006s sys 0.001s
I20260924 00:23:05.927033 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.927165 5224 consensus_queue.cc:237] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.897385 5310 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
W20260924 00:23:05.930605 5120 consensus_meta.cc:287] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.658s user 0.015s sys 0.000s
I20260924 00:23:05.931003 5120 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.933557 5430 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:05.933728 5147 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.936839 2648 raft_consensus.cc:1226] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:05.935093 5261 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:05.941987 2140 raft_consensus.cc:1226] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:05.941936 1993 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:05.976227 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:05.978148 2139 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:05.982090 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:05.982880 2648 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:05.984658 2110 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.985834 2110 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.986824 2110 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:05.939280 5282 raft_consensus.cc:686] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:05.994458 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:05.994529 5282 consensus_queue.cc:237] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:05.995131 1995 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:05.995596 5305 consensus_queue.cc:1046] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
W20260924 00:23:05.996065 5151 consensus_meta.cc:287] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Time spent flushing consensus metadata: real 0.701s user 0.007s sys 0.002s
I20260924 00:23:05.977486 5323 raft_consensus.cc:2726] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:23:05.936482 5360 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:05.973793 5120 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:06.001543 5460 consensus_queue.cc:1046] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:23:06.002698 5460 raft_consensus.cc:2781] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:23:06.003207 5460 raft_consensus.cc:686] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:06.003443 5377 raft_consensus.cc:2781] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:23:06.003598 5400 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:23:06.003871 5352 raft_consensus.cc:3032] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:23:06.004011 5377 raft_consensus.cc:686] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:06.004315 5352 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:23:06.004951 5495 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260924 00:23:06.005296 1995 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:06.006625 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:06.006170 5147 raft_consensus.cc:3032] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:06.007965 5147 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:06.008735 5147 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.010025 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:06.010219 2142 raft_consensus.cc:1226] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:06.011489 5261 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:05.996534 5151 raft_consensus.cc:504] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:06.013741 5464 consensus_meta.cc:287] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.765s user 0.010s sys 0.000s
I20260924 00:23:06.014739 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:06.015353 1989 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:06.015174 5387 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.017493 1992 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260924 00:23:06.018957 1995 raft_consensus.cc:3032] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:23:06.019307 1995 raft_consensus.cc:729] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:06.019979 1995 consensus_queue.cc:260] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.005224 5352 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.029906 2110 leader_election.cc:302] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:06.031701 1995 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:23:05.931398 5351 consensus_meta.cc:287] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.632s user 0.009s sys 0.000s
I20260924 00:23:06.002911 2139 raft_consensus.cc:2454] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 12.
I20260924 00:23:06.042140 2144 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:23:06.041488 1989 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:23:06.051268 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:06.016987 5387 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:06.056667 2614 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:23:06.056942 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 159142us
I20260924 00:23:06.039961 5351 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.055845 2035 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:05.912460 5396 consensus_queue.cc:1046] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.129s
I20260924 00:23:06.031486 5151 leader_election.cc:288] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:06.068331 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.067329 5462 raft_consensus.cc:2781] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader election won for term 12
I20260924 00:23:06.069728 5462 raft_consensus.cc:686] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:06.070591 5462 consensus_queue.cc:237] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.040081 5392 raft_consensus.cc:2781] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election won for term 7
I20260924 00:23:06.046972 5493 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:06.073168 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.074914 2137 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:06.072894 2643 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:06.062942 5430 consensus_queue.cc:1046] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20260924 00:23:06.078513 2648 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:06.079069 5360 raft_consensus.cc:3032] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:06.074605 5499 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.079520 5360 raft_consensus.cc:729] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:06.080932 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.081346 5499 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:23:06.081875 5351 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:06.082073 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.082660 5499 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.083222 5499 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.083523 2035 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:06.084352 5461 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:23:06.084837 5461 raft_consensus.cc:686] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:06.085755 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:06.085626 5461 consensus_queue.cc:237] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.087142 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.088009 5499 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:23:06.088212 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1923) took 3179 ms. Trace:
I20260924 00:23:06.088578 2069 rpcz_store.cc:276] 0924 00:23:02.908722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:04.384096 (+1475374us) service_pool.cc:227] Handling call
0924 00:23:04.940621 (+556525us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:04.941054 (+ 433us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:04.941087 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:04.941727 (+ 640us) log.cc:842] Serialized 26 byte log entry
0924 00:23:05.059207 (+117480us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:23:05.059249 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:05.059323 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:05.252510 (+193187us) raft_consensus.cc:1671] finished
0924 00:23:05.252531 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:06.087988 (+835457us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b5400155ed0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abba09 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:06.088164 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3575,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":1279017216,"thread_start_us":92493,"threads_started":1,"wal-append.queue_time_us":170220}
I20260924 00:23:06.080420 5360 consensus_queue.cc:260] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.046777 5401 consensus_queue.cc:1046] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260924 00:23:06.073453 5407 raft_consensus.cc:2781] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader election won for term 6
I20260924 00:23:06.085546 5127 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:06.099555 1995 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:23:06.094082 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.004096 5460 consensus_queue.cc:237] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.005071 5377 consensus_queue.cc:237] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.066103 5120 consensus_queue.cc:1046] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260924 00:23:06.074036 5400 raft_consensus.cc:686] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:06.122851 5445 raft_consensus.cc:3032] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:06.105818 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.077497 5392 raft_consensus.cc:686] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:06.108116 5360 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:06.109611 5167 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.113547 5407 raft_consensus.cc:686] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:06.114984 5289 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:23:06.116397 2138 raft_consensus.cc:1261] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260924 00:23:06.120847 5342 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 00:23:06.183310 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 77563.5us
W20260924 00:23:06.186123 5270 consensus_meta.cc:287] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.842s user 0.007s sys 0.003s
W20260924 00:23:06.243006 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 48713.1us
I20260924 00:23:06.243505 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.186548 5167 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:06.187106 5266 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.187490 5465 raft_consensus.cc:3032] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:23:06.193243 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:06.194551 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:06.088872 5500 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:06.093999 5493 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.249743 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.270237 1960 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:06.250820 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 52200.8us
I20260924 00:23:06.123282 5445 raft_consensus.cc:729] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:06.106809 5127 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.123170 5380 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.286849 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.290922 5266 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260924 00:23:06.250874 5387 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:06.254362 5465 raft_consensus.cc:729] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:06.326654 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:06.332393 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 45651.2us
I20260924 00:23:06.335963 2136 raft_consensus.cc:1226] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:06.338495 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.340961 5336 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.283968 5342 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:23:06.250885 5270 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:23:06.371452 2138 raft_consensus.cc:1226] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:06.360814 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.372938 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.373106 5430 consensus_queue.cc:1046] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:23:06.373983 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
W20260924 00:23:06.374511 5504 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260924 00:23:06.375473 5310 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:06.375538 1960 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.376173 5310 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260924 00:23:06.376708 5310 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260924 00:23:06.376647 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.377188 5310 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:06.377578 5310 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Advancing to term 12
W20260924 00:23:06.351022 5418 consensus_meta.cc:287] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.588s user 0.008s sys 0.002s
I20260924 00:23:06.273618 5408 consensus_queue.cc:1035] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260924 00:23:06.351908 5387 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.253279 5400 consensus_queue.cc:237] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.357667 5336 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:06.369376 5266 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:23:06.388223 5418 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:23:06.390125 5500 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.389686 5342 raft_consensus.cc:2781] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:23:06.417259 5323 consensus_queue.cc:1046] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:23:06.427289 5323 raft_consensus.cc:3032] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Stepping down as leader of term 10
I20260924 00:23:06.427711 5323 raft_consensus.cc:729] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:06.428419 5369 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:06.355701 5407 consensus_queue.cc:237] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.429474 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:06.411862 5336 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.434494 5336 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:06.435665 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:06.435999 5147 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:06.422757 5127 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:06.436337 2138 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:06.428771 5323 consensus_queue.cc:260] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.437835 2035 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:06.439606 5336 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260924 00:23:06.440129 5498 log.cc:924] Time spent T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.380s user 0.007s sys 0.007s
I20260924 00:23:06.438333 5323 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:05.829246 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:23:06.435686 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:06.440821 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47770 (request call id 2289) took 4031 ms. Trace:
I20260924 00:23:06.441125 1996 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:23:06.443529 5336 raft_consensus.cc:482] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:06.444868 5336 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:06.443466 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:06.446090 5159 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:06.446461 5159 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:06.446791 5159 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:06.449396 5493 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:06.451292 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:06.451730 5270 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775)
I20260924 00:23:06.452239 5270 consensus_queue.cc:1046] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:23:06.453011 5270 raft_consensus.cc:3032] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:06.453341 5270 raft_consensus.cc:729] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:06.441262 2065 rpcz_store.cc:276] 0924 00:23:02.409780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:02.409933 (+ 153us) service_pool.cc:227] Handling call
0924 00:23:02.410301 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:05.317040 (+2906739us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b58003b90b0. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4a37f63 00007fb6d4a9baba 00007fb6d4ab33af 00007fb6d4ab4b4e 00007fb6d4abec89 00007fb6d4abf556 00007fb6d4abbedf 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684
0924 00:23:05.456054 (+139014us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:06.440565 (+984511us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x7b5400305a54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:06.440749 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5703732864}
W20260924 00:23:06.443745 5503 log.cc:924] Time spent T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.384s user 0.000s sys 0.014s
I20260924 00:23:06.345669 5392 consensus_queue.cc:237] T 4c4f367683e243cdb37181c1bdb546ec P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.454149 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:23:06.464723 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 86903.6us
I20260924 00:23:06.369062 5445 consensus_queue.cc:260] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:06.465889 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 60220.1us
W20260924 00:23:06.475718 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 1.29147e+06us
I20260924 00:23:06.480074 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.482297 2109 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.483666 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.484683 5297 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:06.485390 5464 raft_consensus.cc:2726] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:23:06.485432 5297 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.486816 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.487680 5297 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260924 00:23:06.494458 2070 raft_consensus.cc:1226] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:06.496717 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:06.497510 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:06.499599 2067 raft_consensus.cc:3037] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:06.423766 5342 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:06.502761 2066 raft_consensus.cc:1226] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:06.498356 2063 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:06.504549 2139 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:06.500777 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:06.495447 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:06.510737 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:06.515717 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:06.516537 1995 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:06.519688 2062 raft_consensus.cc:1226] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 12. Current term is 13. Ops: []
I20260924 00:23:06.524398 2641 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:06.532416 2068 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:06.454149 5270 consensus_queue.cc:260] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.394375 5465 consensus_queue.cc:260] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.507659 5342 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:23:06.507815 5439 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:06.559056 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:06.562692 5270 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:06.566406 5509 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:06.569365 5380 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:06.545334 5266 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:06.573761 2067 raft_consensus.cc:2454] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
I20260924 00:23:06.548722 5186 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:06.492460 5500 consensus_queue.cc:1046] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260924 00:23:06.599472 1995 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:06.611053 2110 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:06.544029 5159 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.595677 5465 raft_consensus.cc:3037] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:06.570127 5439 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:06.637079 2139 raft_consensus.cc:2396] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
W20260924 00:23:06.641323 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 81136.2us
I20260924 00:23:06.564143 5460 consensus_queue.cc:1035] T cfca69df981d4ab89f1ea62a1d8c1557 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260924 00:23:06.581920 5271 consensus_queue.cc:1046] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260924 00:23:06.625278 5377 raft_consensus.cc:2781] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:23:06.674887 5522 consensus_queue.cc:1035] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260924 00:23:06.609965 5186 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:06.689745 5507 log.cc:924] Time spent T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260924 00:23:06.689947 5186 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:06.691459 5186 consensus_queue.cc:1046] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:23:06.695477 1995 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:23:06.670171 5500 raft_consensus.cc:3032] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:23:06.693629 5377 raft_consensus.cc:686] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:06.637058 5430 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260924 00:23:06.697662 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 2435) took 1281 ms. Trace:
I20260924 00:23:06.719449 2141 rpcz_store.cc:276] 0924 00:23:05.416661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.417265 (+ 604us) service_pool.cc:227] Handling call
0924 00:23:05.417514 (+ 249us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:05.808298 (+390784us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:06.697445 (+889147us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b540025e754. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:06.697609 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":881047296}
I20260924 00:23:06.698623 2643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45594 (request call id 655) took 1213 ms. Trace:
I20260924 00:23:06.721227 2643 rpcz_store.cc:276] 0924 00:23:05.485117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.490760 (+ 5643us) service_pool.cc:227] Handling call
0924 00:23:05.491055 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:06.076832 (+585777us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:06.698378 (+621546us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b540050f854. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:06.698564 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1315015936}
I20260924 00:23:06.700273 2644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45582 (request call id 667) took 1075 ms. Trace:
W20260924 00:23:06.702431 5508 log.cc:924] Time spent T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260924 00:23:06.726464 2141 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260924 00:23:06.669857 5305 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:06.684808 5401 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:23:06.713943 5360 consensus_meta.cc:287] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.538s user 0.010s sys 0.000s
I20260924 00:23:06.747159 2141 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:06.713325 5509 raft_consensus.cc:3032] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Stepping down as leader of term 8
W20260924 00:23:06.755162 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 118975us
I20260924 00:23:06.723500 5500 raft_consensus.cc:729] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
W20260924 00:23:06.759485 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 87711.2us
I20260924 00:23:06.772271 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:06.772972 1987 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:06.774539 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:06.814076 2641 raft_consensus.cc:1261] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260924 00:23:06.816569 2649 raft_consensus.cc:1226] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:06.817502 2649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45568 (request call id 515) took 1451 ms. Trace:
I20260924 00:23:06.817941 2649 rpcz_store.cc:276] 0924 00:23:05.366440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.369121 (+ 2681us) service_pool.cc:227] Handling call
0924 00:23:06.816073 (+1446952us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b5800005a98. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d9a122b0 00007fb6d99fd7f9 00007fb6d99fd2a2 00007fb6d30e3684
0924 00:23:06.816319 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:06.817108 (+ 789us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:06.817438 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2714880}
I20260924 00:23:06.760354 5418 consensus_queue.cc:1046] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260924 00:23:06.821043 5418 consensus_queue.cc:1225] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer 9e79063532384ada8ea8f8205478adcf log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260924 00:23:06.772588 5518 log.cc:924] Time spent T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260924 00:23:06.776706 5366 consensus_queue.cc:1046] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260924 00:23:06.826501 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 628) took 1539 ms. Trace:
I20260924 00:23:06.780246 2036 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
W20260924 00:23:06.921806 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 141685us
I20260924 00:23:06.756004 5509 raft_consensus.cc:729] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:06.742120 5122 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260924 00:23:06.853032 1987 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:06.724309 2644 rpcz_store.cc:276] 0924 00:23:05.624924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.628192 (+ 3268us) service_pool.cc:227] Handling call
0924 00:23:05.745782 (+117590us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:05.847223 (+101441us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:06.699623 (+852400us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b540069e454. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:06.700213 (+ 590us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31605632}
I20260924 00:23:06.745977 5305 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.872236 2643 raft_consensus.cc:1203] T 88e267ffb8c4467b93815c1a9e3d3664 P 9e79063532384ada8ea8f8205478adcf [term 7 LEARNER]: Deduplicated request from leader. Original: 7.4->[7.5-7.5] Dedup: 7.5->[]
I20260924 00:23:06.897357 2145 raft_consensus.cc:1226] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [term 6 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:06.920393 2137 rpcz_store.cc:276] 0924 00:23:05.287421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.287632 (+ 211us) service_pool.cc:227] Handling call
0924 00:23:05.287969 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:06.075342 (+787373us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:06.826235 (+750893us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b540025e254. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:06.826433 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1809444736}
I20260924 00:23:06.871609 5159 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:06.950294 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:06.945642 2137 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8. Ops: []
W20260924 00:23:06.953833 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 178333us
I20260924 00:23:06.812077 5519 consensus_queue.cc:1046] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20260924 00:23:06.749002 5401 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:06.957218 1960 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:06.929522 5439 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:06.966513 2137 raft_consensus.cc:1261] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260924 00:23:06.961388 2646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45568 (request call id 517) took 1577 ms (client timeout 2079 ms). Trace:
I20260924 00:23:06.927208 5271 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:06.929706 5430 raft_consensus.cc:3032] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
W20260924 00:23:06.945832 5488 log.cc:924] Time spent T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 1.225s user 0.008s sys 0.010s
I20260924 00:23:06.963740 2110 leader_election.cc:302] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:23:06.968031 2646 rpcz_store.cc:269] 0924 00:23:05.377507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.378427 (+ 920us) service_pool.cc:227] Handling call
0924 00:23:06.950130 (+1571703us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b5800005a98. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1b5c851 00007fb6d1b5c702 00007fb6daa4cb09 00007fb6d9988730 00007fb6d9a4e79b 00007fb6d9a4e821 00007fb6d99fe1b0 00007fb6d99fda94 00007fb6d30e5c04 00007fb6d30e5b92
0924 00:23:06.954454 (+ 4324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11746176}
I20260924 00:23:06.929210 5266 consensus_queue.cc:1046] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260924 00:23:06.977329 1987 raft_consensus.cc:1261] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260924 00:23:06.856660 5377 consensus_queue.cc:237] T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.010303 2137 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:06.971616 5524 raft_consensus.cc:2741] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260924 00:23:06.974195 5430 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:07.030315 5305 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:07.022233 5266 consensus_queue.cc:1225] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 948356bf66a941ef8249ecbd0ec94f6d log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260924 00:23:07.006148 5483 consensus_queue.cc:1046] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260924 00:23:07.003119 5439 consensus_queue.cc:1046] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260924 00:23:07.038235 5439 consensus_queue.cc:1225] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 948356bf66a941ef8249ecbd0ec94f6d log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260924 00:23:07.038806 5439 raft_consensus.cc:3032] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Stepping down as leader of term 8
I20260924 00:23:07.039173 5439 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:07.014539 5271 raft_consensus.cc:2781] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:23:06.961272 5462 consensus_queue.cc:1035] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260924 00:23:06.965044 5127 consensus_queue.cc:1046] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260924 00:23:06.959146 5528 consensus_queue.cc:1046] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260924 00:23:06.926823 5500 consensus_queue.cc:260] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.024274 5380 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:07.074630 5380 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:23:06.983322 5509 consensus_queue.cc:260] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:23:07.080227 5401 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:07.079779 5439 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:07.081374 5439 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
W20260924 00:23:07.034303 5147 consensus_meta.cc:287] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.598s user 0.012s sys 0.000s
W20260924 00:23:07.034806 5310 consensus_meta.cc:287] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.657s user 0.006s sys 0.004s
I20260924 00:23:07.069869 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:07.073328 5271 raft_consensus.cc:686] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:07.083199 5524 consensus_queue.cc:1035] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:23:07.086587 5305 raft_consensus.cc:2781] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:07.044732 5122 raft_consensus.cc:3032] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 9 LEADER]: Stepping down as leader of term 9
W20260924 00:23:07.087440 5516 log.cc:924] Time spent T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20260924 00:23:07.104574 2143 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 10.
I20260924 00:23:07.069746 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 2394) took 1523 ms. Trace:
I20260924 00:23:07.063462 5460 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:07.133587 5122 raft_consensus.cc:729] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:07.264202 5460 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.264658 5122 consensus_queue.cc:260] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:07.265820 5122 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:07.266019 5542 raft_consensus.cc:3032] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:07.266408 5542 raft_consensus.cc:729] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:07.266772 5460 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:23:07.140470 5342 consensus_meta.cc:287] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.571s user 0.006s sys 0.002s
I20260924 00:23:07.267191 5542 consensus_queue.cc:260] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:07.267761 5342 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:07.135896 5305 raft_consensus.cc:686] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:07.270228 5342 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:07.148869 5271 consensus_queue.cc:237] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.270848 5305 consensus_queue.cc:237] T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:07.151238 5336 consensus_meta.cc:287] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.681s user 0.008s sys 0.002s
I20260924 00:23:07.273259 5336 raft_consensus.cc:504] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.275218 5336 leader_election.cc:288] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:07.115425 5310 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.147388 5380 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
W20260924 00:23:07.163843 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 224624us
W20260924 00:23:07.182315 5465 consensus_meta.cc:287] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.545s user 0.010s sys 0.000s
W20260924 00:23:07.211802 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 288751us
W20260924 00:23:07.224890 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 261190us
I20260924 00:23:07.268579 5542 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:07.282853 5307 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:07.283403 5307 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:07.285851 5313 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260924 00:23:07.277069 5310 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:07.290117 5380 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:07.291181 5396 raft_consensus.cc:3032] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:23:07.290814 5380 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.291613 5396 raft_consensus.cc:729] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:07.292407 5396 consensus_queue.cc:260] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:07.294042 5380 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:07.263940 1988 rpcz_store.cc:276] 0924 00:23:05.546227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.546412 (+ 185us) service_pool.cc:227] Handling call
0924 00:23:05.546843 (+ 431us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:05.547127 (+ 284us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:05.547168 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:05.547707 (+ 539us) log.cc:842] Serialized 26 byte log entry
0924 00:23:05.676506 (+128799us) raft_consensus.cc:1631] Marking committed up to 5
0924 00:23:05.676540 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:05.676604 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:06.833410 (+1156806us) spinlock_profiling.cc:243] Waited 9.7 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:23:07.045723 (+212313us) raft_consensus.cc:1671] finished
0924 00:23:07.045746 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:07.069663 (+ 23917us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22315904,"thread_start_us":128391,"threads_started":1,"wal-append.queue_time_us":168387}
W20260924 00:23:08.451149 2092 debug-util.cc:398] Leaking SignalData structure 0x7b080030e260 after lost signal to thread 812
W20260924 00:23:08.451769 2092 debug-util.cc:398] Leaking SignalData structure 0x7b0800172be0 after lost signal to thread 815
I20260924 00:23:10.254125 5567 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.257458 5567 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.262262 5567 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.262918 5567 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.263679 5567 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.267014 5567 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.267104 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 654) took 2988 ms. Trace:
I20260924 00:23:10.267576 5567 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.267563 2139 rpcz_store.cc:276] 0924 00:23:07.278945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:10.265162 (+2986217us) service_pool.cc:227] Handling call
0924 00:23:10.266025 (+ 863us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:10.266445 (+ 420us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:10.266489 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:10.266551 (+ 62us) raft_consensus.cc:1631] Marking committed up to 9
0924 00:23:10.266580 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:10.266668 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:10.266876 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260924 00:23:10.268556 5567 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.272941 5567 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.273589 5567 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.274261 5567 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.276477 5567 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.276937 5567 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.277638 5567 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.295568 5567 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.298882 2142 raft_consensus.cc:1226] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:23:10.300038 5141 raft_consensus.cc:3032] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:23:10.300484 5141 raft_consensus.cc:729] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:10.302106 5141 consensus_queue.cc:260] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.304183 5141 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:10.306437 5464 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.307947 5464 raft_consensus.cc:504] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.313550 5464 leader_election.cc:288] T 9ff940e15e1849efb194c908520c212f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.314064 5464 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:10.316594 2144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57748 (request call id 2074) took 4786 ms (client timeout 1612 ms). Trace:
I20260924 00:23:10.314720 5464 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:10.317174 2144 rpcz_store.cc:269] 0924 00:23:05.530132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.530337 (+ 205us) service_pool.cc:227] Handling call
0924 00:23:10.316156 (+4785819us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b540025f154. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:10.316368 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1054989568}
I20260924 00:23:10.319092 5464 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.320595 5464 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.321337 5464 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.324018 5464 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:10.338795 5305 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.057s user 0.017s sys 0.047s
W20260924 00:23:10.339466 5305 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.057s user 0.017s sys 0.047s
W20260924 00:23:10.353837 1960 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.378s user 0.017s sys 0.078s
W20260924 00:23:10.354382 1960 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.384s user 0.017s sys 0.078s
W20260924 00:23:10.354993 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 3.3976e+06us
I20260924 00:23:10.357435 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 2436) took 4785 ms. Trace:
I20260924 00:23:10.357986 2136 rpcz_store.cc:276] 0924 00:23:05.572269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.572398 (+ 129us) service_pool.cc:227] Handling call
0924 00:23:05.572762 (+ 364us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:06.336465 (+763703us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:10.356833 (+4020368us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b540021b954. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:10.357200 (+ 367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1754775808}
I20260924 00:23:10.358888 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.359828 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.361089 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.362187 1996 raft_consensus.cc:2454] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 11.
I20260924 00:23:10.362596 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.357463 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.369374 5186 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.370345 5186 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.376411 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.377444 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.373564 5186 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.377485 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.379240 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.379619 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.380249 1989 raft_consensus.cc:2396] T cfca69df981d4ab89f1ea62a1d8c1557 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260924 00:23:10.381986 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.384613 5270 raft_consensus.cc:3032] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:23:10.385066 5270 raft_consensus.cc:729] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:10.385807 5137 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:10.386612 5485 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.386797 5137 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.387495 5485 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.389400 5137 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.389894 5485 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.385877 5305 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:10.390858 5305 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.393069 5464 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.393347 5305 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.394129 5311 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.393914 5464 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.394918 5311 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.396286 5464 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.377076 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.400272 5485 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.418743 5485 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.419916 5461 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.420722 5461 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.400436 5464 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.422001 5464 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.423592 5461 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:10.423833 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.501s user 0.028s sys 0.024s
W20260924 00:23:10.424248 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.502s user 0.028s sys 0.024s
I20260924 00:23:10.424120 5461 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.413904 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60184 (request call id 2453) took 4055 ms. Trace:
W20260924 00:23:10.424655 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 3.50188e+06us
I20260924 00:23:10.401243 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:10.416145 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.427659 5485 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.443614 5485 raft_consensus.cc:482] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.444341 5485 raft_consensus.cc:504] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.424000 5464 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:10.400518 2110 connection.cc:232] RPC call timeout handler was delayed by -1.6849420860000919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20260924 00:23:10.447083 2110 connection.cc:232] RPC call timeout handler was delayed by -1.6371152340000208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1613s for a response
W20260924 00:23:10.447484 2110 connection.cc:232] RPC call timeout handler was delayed by -1.6327420860000075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260924 00:23:10.447778 2110 connection.cc:232] RPC call timeout handler was delayed by -1.6021420860001854s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
I20260924 00:23:10.446580 5485 leader_election.cc:288] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.425122 2145 rpcz_store.cc:276] 0924 00:23:06.358592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:06.358756 (+ 164us) service_pool.cc:227] Handling call
0924 00:23:06.359046 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:06.938118 (+579072us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:10.413350 (+3475232us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b5400330ed4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:10.413701 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1237507456}
I20260924 00:23:10.397073 5311 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.386613 5270 consensus_queue.cc:260] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.436206 1995 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:10.403467 1960 leader_election.cc:302] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:10.425172 5461 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.441525 5305 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:10.441673 2037 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.778s user 0.008s sys 0.023s
I20260924 00:23:10.442265 5380 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:10.440876 5119 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.461301 5380 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:10.461469 5119 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:10.460052 2037 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.796s user 0.009s sys 0.023s
I20260924 00:23:10.463740 5305 raft_consensus.cc:504] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.466991 5305 leader_election.cc:288] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:10.467309 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 64065.5us
I20260924 00:23:10.467572 5305 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:10.468612 5461 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.468356 5305 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.476617 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.477650 2136 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 3.
I20260924 00:23:10.478863 2062 raft_consensus.cc:1226] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:10.483839 5380 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.484177 5119 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.484654 5380 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.484771 5119 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.485797 5119 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.463143 5360 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.487319 5360 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.485533 5380 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.489702 5547 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.490424 5547 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.491003 5380 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.491293 5305 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.497102 5307 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.492693 5119 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.500032 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.500859 5369 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.501631 5369 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.502480 5545 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.503628 5369 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.503474 5545 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.509075 5301 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:10.510105 5301 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.510463 5360 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.512298 5301 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.512692 5545 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.513211 1987 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:23:10.459491 2036 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:10.518285 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:10.530342 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:10.530508 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:10.531286 2064 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:10.531591 2068 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:23:10.538431 2065 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:10.539717 5242 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:10.541226 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.541730 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:10.541971 5439 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.542569 5439 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.544909 5439 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.546942 5439 consensus_queue.cc:1046] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:10.547477 5310 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.548193 5310 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.543068 5242 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:10.553025 2613 connection.cc:232] RPC call timeout handler was delayed by -3.1636826039998596s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20260924 00:23:10.553583 2613 connection.cc:232] RPC call timeout handler was delayed by -3.1096826039999996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
I20260924 00:23:10.553720 5242 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:10.553975 2613 connection.cc:232] RPC call timeout handler was delayed by -3.0965197789998911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
W20260924 00:23:10.554598 2613 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.203.2:39591 timed out after 1.898s (SENT) [suppressed 20 similar messages]
I20260924 00:23:10.492496 5547 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.548117 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47766 (request call id 1947) took 4048 ms. Trace:
I20260924 00:23:10.558092 2065 rpcz_store.cc:276] 0924 00:23:06.493200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:06.494345 (+ 1145us) service_pool.cc:227] Handling call
0924 00:23:06.494636 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:10.539020 (+4044384us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:10.540770 (+ 1750us) spinlock_profiling.cc:243] Waited 4.04 s on lock 0x7b54003f8ed4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:10.541059 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9298754944}
I20260924 00:23:10.530812 2067 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
I20260924 00:23:10.570233 5547 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:10.571583 5547 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.576715 2064 raft_consensus.cc:2454] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 12.
I20260924 00:23:10.578845 5396 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:10.580235 5352 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:10.583544 5310 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.584249 5310 raft_consensus.cc:3032] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:10.586050 5310 raft_consensus.cc:729] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:10.587235 5310 consensus_queue.cc:260] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:10.591228 5491 log.cc:924] Time spent T ba683df200644d888896fb84dbc748e8 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 4.875s user 0.032s sys 0.071s
I20260924 00:23:10.592382 2643 raft_consensus.cc:1226] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:10.593303 5547 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:10.605599 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 148851us
I20260924 00:23:10.553607 5439 raft_consensus.cc:2726] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:23:10.595513 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:10.599496 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.611043 2036 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:10.608112 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:10.612577 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:10.609069 5310 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:10.614245 5525 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:23:10.614269 2036 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.594162 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:10.615448 5525 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:10.622278 5533 raft_consensus.cc:3032] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:23:10.622786 5533 raft_consensus.cc:729] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:10.625751 5266 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:10.627065 5266 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.630370 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42938 (request call id 755) took 4227 ms. Trace:
I20260924 00:23:10.630780 1992 rpcz_store.cc:276] 0924 00:23:06.402543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:06.402694 (+ 151us) service_pool.cc:227] Handling call
0924 00:23:06.403071 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:06.420593 (+ 17522us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:06.420639 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:06.420695 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0924 00:23:06.420739 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:10.629935 (+4209196us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b540039c454. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:10.630005 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:10.630180 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39737216}
I20260924 00:23:10.631420 5266 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.631847 5525 raft_consensus.cc:2726] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:23:10.633196 2036 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.608814 5270 raft_consensus.cc:3037] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:10.622810 2068 raft_consensus.cc:2454] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:23:10.600421 5519 raft_consensus.cc:3032] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 12 LEADER]: Stepping down as leader of term 12
I20260924 00:23:10.643090 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.644774 5552 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:10.646013 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:10.637187 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 82646.4us
W20260924 00:23:10.648005 2613 connection.cc:232] RPC call timeout handler was delayed by -3.0087091570001121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
I20260924 00:23:10.650179 5439 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.650913 5412 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 00:23:10.651227 5439 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:10.653692 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 75941.4us
I20260924 00:23:10.654184 5547 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.654863 5547 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.656337 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.656623 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.657358 2142 raft_consensus.cc:2396] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260924 00:23:10.658352 2613 connection.cc:232] RPC call timeout handler was delayed by -2.855882604000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19710000000000003s for a response
W20260924 00:23:10.658787 2613 connection.cc:232] RPC call timeout handler was delayed by -2.6579442449999533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
I20260924 00:23:10.662827 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.643805 5519 raft_consensus.cc:729] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:10.665762 5529 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:10.624029 5533 consensus_queue.cc:260] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.666711 5529 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.667429 5533 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:10.643303 5266 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:10.670251 5569 raft_consensus.cc:482] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.670573 5529 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.669152 2140 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:23:10.656879 5547 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.672712 5392 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.673442 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.671348 5266 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.670965 5569 raft_consensus.cc:504] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.681492 5283 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:10.681181 5392 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.666714 5519 consensus_queue.cc:260] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.687624 1992 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 11. Current term is 12. Ops: []
W20260924 00:23:10.687769 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 54680.4us
W20260924 00:23:10.689805 2613 connection.cc:232] RPC call timeout handler was delayed by -1.7423510149999402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260924 00:23:10.690001 5391 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.690579 5391 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.691288 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.681234 5529 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:10.681259 2145 raft_consensus.cc:2396] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260924 00:23:10.684079 5266 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.684408 5569 leader_election.cc:288] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.682214 5283 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.692025 1987 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:23:10.693596 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.693843 5391 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.695268 5493 consensus_queue.cc:1046] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:10.687798 5519 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Advancing to term 13
I20260924 00:23:10.689020 5439 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.699703 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.699714 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.701185 5283 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.703358 1989 raft_consensus.cc:1261] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:23:10.705566 2036 leader_election.cc:302] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.706642 2145 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:23:10.708109 1995 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260924 00:23:10.712826 5493 raft_consensus.cc:2726] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:23:10.713699 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.713900 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.705360 5529 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:10.716922 2036 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.717163 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.717962 2145 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:23:10.718158 5529 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.718818 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.724766 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.717957 5493 raft_consensus.cc:2726] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:23:10.726681 5493 raft_consensus.cc:2781] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260924 00:23:10.731325 5493 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:10.729012 5529 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:10.737879 5539 raft_consensus.cc:482] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.736106 5493 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:10.738529 5539 raft_consensus.cc:504] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.742136 5569 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:10.743211 5539 leader_election.cc:288] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.743639 5569 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.743986 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:23:10.745301 2143 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39164 (request call id 640) took 4735 ms (client timeout 1629 ms). Trace:
I20260924 00:23:10.745645 2145 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 10.
W20260924 00:23:10.745736 2143 rpcz_store.cc:269] 0924 00:23:06.009394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:06.009801 (+ 407us) service_pool.cc:227] Handling call
0924 00:23:10.744940 (+4735139us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540021be54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:10.745093 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2480195968}
I20260924 00:23:10.746287 5569 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.753310 5529 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.755110 5500 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:10.756461 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.755750 5529 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.757189 1995 raft_consensus.cc:2454] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:23:10.758159 5529 raft_consensus.cc:482] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.759779 5529 raft_consensus.cc:504] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.762933 5529 leader_election.cc:288] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:10.764089 1960 connection.cc:232] RPC call timeout handler was delayed by -2.3795072290001826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
W20260924 00:23:10.764415 1960 connection.cc:232] RPC call timeout handler was delayed by -2.0321072290000757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21840000000000004s for a response
I20260924 00:23:10.766564 1992 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 13.
I20260924 00:23:10.768183 2036 leader_election.cc:302] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.768725 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:10.769953 2036 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.770614 5519 raft_consensus.cc:2726] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:23:10.771638 2036 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.771960 5539 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:23:10.772562 5552 raft_consensus.cc:2726] T ba683df200644d888896fb84dbc748e8 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:10.772331 5519 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:10.765331 5569 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.773343 5552 raft_consensus.cc:2726] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:23:10.774112 5552 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260924 00:23:10.773883 5569 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.774116 5519 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:10.776233 5569 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.776711 5509 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:10.777038 5529 raft_consensus.cc:3032] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:10.777897 5529 raft_consensus.cc:729] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:10.778020 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.777056 5392 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:10.778395 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.778712 1993 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:10.779037 1995 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:23:10.779234 5529 consensus_queue.cc:260] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.780422 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.780747 5529 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:10.781185 1995 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
I20260924 00:23:10.798890 5526 consensus_queue.cc:1046] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260924 00:23:10.801647 5290 raft_consensus.cc:3032] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:23:10.802052 5290 raft_consensus.cc:729] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:10.803104 5290 consensus_queue.cc:260] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.804071 2036 leader_election.cc:302] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.804523 5290 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:23:10.805076 5523 raft_consensus.cc:2726] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260924 00:23:10.781994 5493 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.809634 5493 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.818288 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.819144 1993 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:23:10.820271 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.821929 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.822690 1996 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:10.816555 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:10.825832 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.836820 5451 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:10.837387 5451 raft_consensus.cc:482] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:10.843497 5520 raft_consensus.cc:3032] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:10.844228 5520 raft_consensus.cc:729] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:10.844359 1960 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:10.837805 5451 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:10.853700 1990 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:10.846249 5519 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:23:10.859639 5301 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:23:10.863083 2035 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.893s user 0.037s sys 0.079s
W20260924 00:23:10.867183 2035 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.897s user 0.037s sys 0.079s
W20260924 00:23:10.868296 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 3.89824e+06us
I20260924 00:23:10.868374 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.869407 1995 raft_consensus.cc:2454] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 4.
I20260924 00:23:10.868433 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.868520 5301 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:23:10.873401 5186 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.873160 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:10.874118 5186 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:10.878036 5550 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.560s user 0.033s sys 0.072s
I20260924 00:23:10.879824 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:10.885221 2068 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
I20260924 00:23:10.887749 5430 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.890822 5391 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:10.891534 5391 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.897572 2138 raft_consensus.cc:1261] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:23:10.899212 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 655) took 3620 ms. Trace:
W20260924 00:23:10.901377 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 97382.4us
I20260924 00:23:10.902345 2036 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:10.866434 5520 consensus_queue.cc:260] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:10.866186 5519 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.903839 1988 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:10.904388 5401 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:10.899614 2138 rpcz_store.cc:276] 0924 00:23:07.279244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:07.289451 (+ 10207us) service_pool.cc:227] Handling call
0924 00:23:10.896781 (+3607330us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:10.897993 (+ 1212us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:10.898977 (+ 984us) spinlock_profiling.cc:243] Waited 3.61 s on lock 0x7b5400240750. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abba09 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:10.899154 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8295281024}
I20260924 00:23:10.902879 5391 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:10.905869 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.879079 5500 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260924 00:23:10.911676 2140 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39164 (request call id 639) took 4932 ms (client timeout 1971 ms). Trace:
I20260924 00:23:10.910527 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.905602 5430 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:10.913394 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:10.912989 5520 raft_consensus.cc:3037] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:10.917999 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:10.913591 5550 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.596s user 0.034s sys 0.074s
I20260924 00:23:10.931133 5186 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:10.927276 2068 raft_consensus.cc:2454] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
W20260924 00:23:10.918083 2109 connection.cc:232] RPC call timeout handler was delayed by -3.5606978589999017s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
I20260924 00:23:10.918284 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:10.910701 5519 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:23:10.915436 2140 rpcz_store.cc:269] 0924 00:23:05.979774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:05.980003 (+ 229us) service_pool.cc:227] Handling call
0924 00:23:10.896610 (+4916607us) spinlock_profiling.cc:243] Waited 4.63 s on lock 0x7b5400240754. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:10.911597 (+ 14987us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10657248640}
I20260924 00:23:10.917976 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.937780 2613 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:23:10.937366 2109 connection.cc:232] RPC call timeout handler was delayed by -3.4905280299999504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2397s for a response
I20260924 00:23:10.919472 5451 raft_consensus.cc:504] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:10.937968 2646 raft_consensus.cc:2454] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
W20260924 00:23:10.956990 2109 connection.cc:232] RPC call timeout handler was delayed by -3.4869225189997906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20260924 00:23:10.941907 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:10.957629 2109 connection.cc:232] RPC call timeout handler was delayed by -3.4833122879997518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
I20260924 00:23:10.942657 2064 raft_consensus.cc:1226] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 12. Current term is 13. Ops: []
I20260924 00:23:10.961184 5282 raft_consensus.cc:2781] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:23:10.961625 5282 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:10.961963 5282 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:10.963100 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.963557 2613 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd
I20260924 00:23:10.917843 5500 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:10.966006 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.966213 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:10.969885 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:10.967042 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 64855.5us
I20260924 00:23:10.975620 2062 raft_consensus.cc:2454] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:23:10.981737 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:10.986860 2643 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 10: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
W20260924 00:23:10.986748 2109 connection.cc:232] RPC call timeout handler was delayed by -3.1284551799999463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260924 00:23:10.987672 2109 connection.cc:232] RPC call timeout handler was delayed by -2.973245555999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26070000000000004s for a response
I20260924 00:23:10.995916 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.973527 5500 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:10.997794 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:10.999123 5465 raft_consensus.cc:482] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:10.999832 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.001822 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.003105 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.004191 2068 raft_consensus.cc:1226] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:11.004026 1989 raft_consensus.cc:2454] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:11.007262 1994 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 9: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.006304 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.005330 2064 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:11.010623 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:11.014108 2109 connection.cc:232] RPC call timeout handler was delayed by -2.86924205299988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20260924 00:23:11.014600 2109 connection.cc:232] RPC call timeout handler was delayed by -2.8679826119998779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
W20260924 00:23:11.014869 2109 connection.cc:232] RPC call timeout handler was delayed by -2.8511017390001143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260924 00:23:11.015046 2109 connection.cc:232] RPC call timeout handler was delayed by -2.7677801139998337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
I20260924 00:23:10.988420 5464 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:11.018124 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:11.020282 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 57858.1us
I20260924 00:23:11.020190 2062 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 11. Current term is 12. Ops: []
W20260924 00:23:11.021560 2109 connection.cc:232] RPC call timeout handler was delayed by -2.2889773989998048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
I20260924 00:23:11.023890 5342 consensus_queue.cc:1046] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:11.024727 5342 consensus_queue.cc:1225] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 9e79063532384ada8ea8f8205478adcf log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260924 00:23:11.024926 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.025568 5349 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:11.025668 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.026484 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.022110 2067 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:23:11.027220 2068 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:23:11.027601 2643 raft_consensus.cc:2379] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:23:11.025934 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:11.029575 2109 connection.cc:232] RPC call timeout handler was delayed by -2.14851608999993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1613s for a response
W20260924 00:23:11.030361 2109 connection.cc:232] RPC call timeout handler was delayed by -2.1380967439997676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
I20260924 00:23:11.019356 5119 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:11.031245 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.026443 2069 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
W20260924 00:23:11.030690 2109 connection.cc:232] RPC call timeout handler was delayed by -2.1259439969999221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
I20260924 00:23:11.033811 1960 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.036020 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:11.032542 2070 raft_consensus.cc:2454] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:23:11.038553 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.039258 2064 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:23:11.032454 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.061968 5434 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:11.061736 5342 consensus_queue.cc:1046] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:23:11.031777 5451 leader_election.cc:288] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:11.066218 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 79604.4us
I20260924 00:23:11.067004 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.037036 5430 raft_consensus.cc:482] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:11.069731 1994 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260924 00:23:11.069909 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.070822 2070 raft_consensus.cc:2396] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260924 00:23:11.068104 2066 raft_consensus.cc:2454] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 2.
I20260924 00:23:11.017868 5465 raft_consensus.cc:504] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.062929 5456 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 00:23:11.029800 5464 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.069300 5186 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:11.076682 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.062575 5119 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.026177 5349 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.041381 5290 consensus_queue.cc:1035] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260924 00:23:11.088987 2070 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 4.
I20260924 00:23:11.075724 5434 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:11.074600 5412 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.091732 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.092276 1960 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
W20260924 00:23:11.095716 5271 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.800s user 0.028s sys 0.021s
I20260924 00:23:11.084414 5430 raft_consensus.cc:504] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.098111 5430 leader_election.cc:288] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.099303 5447 raft_consensus.cc:3032] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:11.099730 5447 raft_consensus.cc:729] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:11.085632 5141 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.109124 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:11.111162 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.112699 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.102561 5342 consensus_queue.cc:1046] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20260924 00:23:11.113860 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.105746 5141 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.114606 5521 raft_consensus.cc:2781] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader election won for term 12
I20260924 00:23:11.115376 1989 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:23:11.102123 5465 leader_election.cc:288] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.119590 5445 raft_consensus.cc:3037] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.100666 5447 consensus_queue.cc:260] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.121685 1987 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:11.123243 1960 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:11.132186 1960 leader_election.cc:302] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:11.117285 5456 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260924 00:23:11.133697 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.122313 5521 raft_consensus.cc:686] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [term 12 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:11.135921 1960 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.138002 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.129453 5528 raft_consensus.cc:2781] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:11.141008 2070 raft_consensus.cc:2379] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 10: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
W20260924 00:23:11.135114 5271 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.862s user 0.028s sys 0.021s
I20260924 00:23:11.136480 5529 raft_consensus.cc:2726] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:11.143015 5349 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:11.143242 5499 raft_consensus.cc:2781] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:23:11.100668 5434 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:11.143642 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 78925.7us
I20260924 00:23:11.133239 5141 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.110572 5412 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.146646 5499 raft_consensus.cc:686] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:11.136376 5342 raft_consensus.cc:2726] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:11.135192 5456 raft_consensus.cc:482] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:11.147130 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.147581 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.151767 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.149772 2649 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:11.155835 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.133773 5430 consensus_queue.cc:1035] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260924 00:23:11.156129 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.158363 2062 raft_consensus.cc:1226] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260924 00:23:11.156549 5569 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:23:11.134518 5465 consensus_queue.cc:1046] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260924 00:23:11.151762 2644 raft_consensus.cc:1226] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:11.150333 5464 leader_election.cc:288] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:11.154848 5456 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:11.150209 5528 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:11.159560 5439 raft_consensus.cc:2781] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:11.158706 5119 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:11.175405 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.178735 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:11.179844 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08000f1740 after lost signal to thread 1928
I20260924 00:23:11.174383 5447 raft_consensus.cc:3037] T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.181124 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.182001 1960 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.182382 5143 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:23:11.183039 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.183194 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.183393 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.183990 2140 raft_consensus.cc:2379] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 10: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:23:11.183909 5266 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.184389 5529 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.184502 5266 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:11.184937 5266 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.173869 5528 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.184860 5529 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:11.194003 5529 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.187837 2143 raft_consensus.cc:2379] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 8: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:11.184186 2136 raft_consensus.cc:2379] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:11.188582 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.166744 5509 raft_consensus.cc:2781] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.166361 5569 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.190634 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.195326 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f6c4b9f5c44d0b3856a8d131557cd" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.152424 5401 consensus_queue.cc:1046] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260924 00:23:11.198923 1960 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.183079 5143 consensus_queue.cc:1225] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 948356bf66a941ef8249ecbd0ec94f6d log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260924 00:23:11.200183 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.201334 2136 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 7.
I20260924 00:23:11.175256 5439 raft_consensus.cc:482] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:11.200148 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.206290 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.207294 2139 raft_consensus.cc:2396] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
I20260924 00:23:11.207505 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.208889 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:11.209774 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.210724 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.197219 5271 raft_consensus.cc:2781] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.211321 2646 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 3.
I20260924 00:23:11.205037 2070 raft_consensus.cc:1226] T aa5e2f99029440bd9d42ec2a01200d4a P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 10. Current term is 11. Ops: []
I20260924 00:23:11.205292 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.204546 5521 consensus_queue.cc:237] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.198356 5509 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:11.215601 5547 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.216454 2069 raft_consensus.cc:1226] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:11.200547 5569 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:11.217803 2142 raft_consensus.cc:2362] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5. Re-sending same reply.
I20260924 00:23:11.214269 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.203480 5342 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:11.221833 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.204996 5439 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:11.219048 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:11.224565 1987 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:23:11.225875 2143 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:11.218181 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.223295 5547 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:11.209285 5143 raft_consensus.cc:2781] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:23:11.231096 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed845c50caec4187b2daed75db23bfd8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.228645 2142 raft_consensus.cc:1226] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:11.231936 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.213634 5310 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:23:11.230285 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.232440 2646 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:11.234694 2140 raft_consensus.cc:2379] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:11.213229 5271 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:11.222124 5509 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.237833 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.248386 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.230436 5547 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.226495 5342 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:11.251420 2062 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:23:11.230600 5282 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:11.255451 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08000fe680 after lost signal to thread 2020
I20260924 00:23:11.231379 5143 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.234896 5310 raft_consensus.cc:482] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.238612 5271 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
W20260924 00:23:11.259184 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08000fe900 after lost signal to thread 2021
I20260924 00:23:11.263109 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.259301 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.262059 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.264786 2145 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
W20260924 00:23:11.266860 2092 debug-util.cc:398] Leaking SignalData structure 0x7b0800332140 after lost signal to thread 2095
W20260924 00:23:11.267171 2092 debug-util.cc:398] Leaking SignalData structure 0x7b080032b500 after lost signal to thread 2096
I20260924 00:23:11.255339 5282 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 11 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.266387 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.271174 5412 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.255383 1890 catalog_manager.cc:5495] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260924 00:23:11.272289 5465 raft_consensus.cc:3032] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:11.261409 5310 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Advancing to term 5
W20260924 00:23:11.273339 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08003b1640 after lost signal to thread 2169
I20260924 00:23:11.273762 5544 raft_consensus.cc:3032] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:11.276041 5567 raft_consensus.cc:3032] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:11.237274 5186 consensus_queue.cc:1046] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20260924 00:23:11.256196 5342 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:11.261122 5143 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:23:11.287505 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
W20260924 00:23:11.291090 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08001b61a0 after lost signal to thread 2170
I20260924 00:23:11.287957 1960 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.285923 5567 raft_consensus.cc:729] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:11.267583 5539 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:11.295032 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:11.296401 2092 debug-util.cc:398] Leaking SignalData structure 0x7b0800354e00 after lost signal to thread 2673
I20260924 00:23:11.296752 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
W20260924 00:23:11.297799 2092 debug-util.cc:398] Leaking SignalData structure 0x7b0800399b20 after lost signal to thread 2675
I20260924 00:23:11.298161 2649 raft_consensus.cc:2379] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 10: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
W20260924 00:23:11.298352 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08003c19e0 after lost signal to thread 5218
I20260924 00:23:11.301447 1960 leader_election.cc:302] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:11.303017 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08003bfc40 after lost signal to thread 5504
I20260924 00:23:11.304598 5387 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.309264 5526 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 00:23:11.311028 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.281009 5465 raft_consensus.cc:729] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:11.228042 5499 consensus_queue.cc:237] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.286111 5544 raft_consensus.cc:729] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:11.321616 1890 catalog_manager.cc:5495] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: HEALTHY } } }
I20260924 00:23:11.300146 5539 raft_consensus.cc:482] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:11.349524 2643 raft_consensus.cc:1226] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:11.349864 1890 catalog_manager.cc:5495] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 6 to 8, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 8 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260924 00:23:11.310752 5533 raft_consensus.cc:3032] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:11.352509 1890 catalog_manager.cc:5495] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd reported cstate change: term changed from 7 to 9, leader changed from 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2) to 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1). New cstate: current_term: 9 leader_uuid: "3eb374c9744b4c7e92dc749aece177dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
W20260924 00:23:11.354547 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 53176.3us
I20260924 00:23:11.357455 1960 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:11.358547 2649 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
W20260924 00:23:11.361227 1893 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33246 (request call id 304) took 11601 ms (client timeout 15000 ms). Trace:
I20260924 00:23:11.361366 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.367573 2139 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:23:11.366886 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.355307 5539 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.379528 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.358192 5533 raft_consensus.cc:729] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:11.369412 5119 consensus_queue.cc:1046] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260924 00:23:11.372581 5580 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.387176 5580 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:11.387586 5580 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.366263 5519 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.391016 5519 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.379158 5567 consensus_queue.cc:260] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.393550 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.394482 5567 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.397605 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.398247 2063 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.400676 5301 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:11.400550 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.402216 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.403035 2062 raft_consensus.cc:2396] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260924 00:23:11.403833 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.404505 2062 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 9: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.405269 5389 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260924 00:23:11.406687 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:11.407305 2141 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.374188 5447 raft_consensus.cc:2726] T 217f6c4b9f5c44d0b3856a8d131557cd P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:11.408576 5580 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:11.409848 5544 consensus_queue.cc:260] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.411026 5580 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.411224 5544 raft_consensus.cc:3037] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.413337 5186 raft_consensus.cc:2726] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260924 00:23:11.387470 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.422726 1990 raft_consensus.cc:2379] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:11.423671 2066 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:11.420665 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.425262 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.428097 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.428292 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.428431 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.428885 2139 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
I20260924 00:23:11.429181 2066 raft_consensus.cc:2379] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 13: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
W20260924 00:23:11.439424 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 82045.2us
I20260924 00:23:11.441045 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.445969 1993 raft_consensus.cc:1226] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:23:11.443606 2646 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:23:11.392654 5509 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.453573 1960 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:11.454383 2062 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 9: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.457949 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.458312 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.459357 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:11.460837 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff940e15e1849efb194c908520c212f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.462157 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.460104 1988 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:11.443202 5392 raft_consensus.cc:3032] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Stepping down as leader of term 11
I20260924 00:23:11.469358 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.471326 2649 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 3.
W20260924 00:23:11.472265 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 44759.3us
I20260924 00:23:11.472724 2065 raft_consensus.cc:2396] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260924 00:23:11.468447 2143 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:23:11.468478 2068 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:11.474336 2138 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:11.474790 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.478178 2070 raft_consensus.cc:2454] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:11.477792 2137 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:23:11.480886 2066 raft_consensus.cc:2454] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:11.481886 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.484371 2064 raft_consensus.cc:1261] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260924 00:23:11.485286 2110 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:23:11.485387 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 45209.9us
I20260924 00:23:11.471943 5583 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
I20260924 00:23:11.486990 2062 raft_consensus.cc:2379] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.451560 5266 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.484758 2064 pending_rounds.cc:76] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 4
I20260924 00:23:11.422580 5465 consensus_queue.cc:260] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.470558 5392 raft_consensus.cc:729] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:11.489537 2064 pending_rounds.cc:97] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260924 00:23:11.489370 2646 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:11.488374 5159 consensus_queue.cc:1035] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260924 00:23:11.490650 5127 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260924 00:23:11.493310 5311 raft_consensus.cc:2781] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260924 00:23:11.495810 5520 consensus_meta.cc:287] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.569s user 0.011s sys 0.000s
I20260924 00:23:11.488438 5394 raft_consensus.cc:482] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:11.497203 2614 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:11.484531 5533 consensus_queue.cc:260] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.498847 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.499524 2110 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:11.502499 5465 raft_consensus.cc:3037] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.504680 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:11.507781 2063 raft_consensus.cc:2454] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:11.521976 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.510116 1960 leader_election.cc:302] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:11.511880 2064 raft_consensus.cc:2866] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:23:11.544627 5584 raft_consensus.cc:2726] T ed845c50caec4187b2daed75db23bfd8 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260924 00:23:11.546610 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.547398 1960 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:11.483647 5569 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.516324 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.520116 2062 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 11. Current term is 12. Ops: []
I20260924 00:23:11.522907 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4917c5d11b34f7aaa5cbdcf98709f22" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.524610 5394 raft_consensus.cc:504] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.526852 5342 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.529417 5408 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.529595 5392 consensus_queue.cc:260] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.513613 5583 raft_consensus.cc:686] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:11.556914 2067 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:11.511652 5311 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.532079 2110 leader_election.cc:302] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:11.532196 5349 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260924 00:23:11.520998 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 136640us
I20260924 00:23:11.529467 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.534801 2063 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
W20260924 00:23:11.361622 1893 rpcz_store.cc:269] 0924 00:22:59.732749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:22:59.732890 (+ 141us) service_pool.cc:227] Handling call
0924 00:23:11.321879 (+11588989us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0924 00:23:02.259047 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 00:23:02.321997 (+ 62950us) write_op.cc:432] Acquiring schema lock in shared mode
0924 00:23:02.322032 (+ 35us) write_op.cc:435] Acquired schema lock
0924 00:23:02.322048 (+ 16us) tablet.cc:660] Decoding operations
0924 00:23:02.335965 (+ 13917us) write_op.cc:620] Acquiring the partition lock for write op
0924 00:23:02.336049 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0924 00:23:02.336073 (+ 24us) tablet.cc:683] Acquiring locks for 4 operations
0924 00:23:02.336447 (+ 374us) tablet.cc:699] Row locks acquired
0924 00:23:02.336465 (+ 18us) write_op.cc:260] PREPARE: finished
0924 00:23:02.336601 (+ 136us) write_op.cc:270] Start()
0924 00:23:02.336707 (+ 106us) write_op.cc:276] Timestamp: P: 1790209382336557 usec, L: 0
0924 00:23:02.336728 (+ 21us) op_driver.cc:331] REPLICATION: starting
0924 00:23:02.340413 (+ 3685us) log.cc:842] Serialized 1677 byte log entry
0924 00:23:04.954454 (+2614041us) spinlock_profiling.cc:243] Waited 6.8 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d1b765fe 00007fb6da9ada39 00007fb6d1b938bb 00007fb6d1b91cbf 00007fb6d1b93e22 00007fb6d473d5bb
0924 00:23:05.331222 (+376768us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b1800342e20. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6daa338b1 00007fb6daa40661 00007fb6d84cc8a8 00007fb6d84cea78 00007fb6d84cea28 00007fb6d84ce9a8 00007fb6d84ce963 00007fb6d84cdc4c 00007fb6daac5206 00007fb6daabd808 00007fb6d4ac8170
0924 00:23:05.331277 (+ 55us) op_driver.cc:444] REPLICATION: finished
0924 00:23:05.536833 (+205556us) write_op.cc:301] APPLY: starting
0924 00:23:05.547341 (+ 10508us) tablet.cc:1365] starting BulkCheckPresence
0924 00:23:05.571557 (+ 24216us) tablet.cc:1368] finished BulkCheckPresence
0924 00:23:05.576275 (+ 4718us) tablet.cc:1370] starting ApplyRowOperation cycle
0924 00:23:05.608000 (+ 31725us) tablet.cc:1381] finished ApplyRowOperation cycle
0924 00:23:05.608197 (+ 197us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0924 00:23:05.608237 (+ 40us) write_op.cc:312] APPLY: finished
0924 00:23:05.608432 (+ 195us) log.cc:842] Serialized 41 byte log entry
0924 00:23:05.608588 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:23:05.608676 (+ 88us) write_op.cc:454] Released schema lock
0924 00:23:05.608718 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0924 00:23:05.609027 (+ 309us) write_op.cc:489] Releasing partition, row and schema locks
0924 00:23:05.609056 (+ 29us) write_op.cc:454] Released schema lock
0924 00:23:05.837724 (+228668us) spinlock_profiling.cc:243] Waited 92.7 ms on lock 0x7b5400002dd4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ab9689 00007fb6d4a57804 00007fb6d4a577b3 00007fb6d4a57728 00007fb6d4a576e3 00007fb6d4a569cc 00007fb6d4a80a16 00007fb6d4a685cb 00007fb6d4a52c8d 00007fb6d4a56665
Metrics: {"mutex_wait_us":1601657,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":184857,"apply.run_cpu_time_us":2642,"apply.run_wall_time_us":86253,"num_ops":4,"prepare.queue_time_us":321935,"prepare.run_cpu_time_us":53144,"prepare.run_wall_time_us":2844073,"raft.queue_time_us":7469,"raft.run_cpu_time_us":62710,"raft.run_wall_time_us":856714,"replication_time_us":2760753,"spinlock_wait_cycles":236939904,"thread_start_us":2947820,"threads_started":4,"wal-append.queue_time_us":2630452,"wal-append.run_cpu_time_us":9044,"wal-append.run_wall_time_us":1644107}]]}
I20260924 00:23:11.537881 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.538559 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.540421 2035 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.540656 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:11.548578 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:23:11.617650 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 85656.7us
W20260924 00:23:11.611243 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 63949.9us
I20260924 00:23:11.623485 2110 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:11.624269 2110 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:11.620419 2066 raft_consensus.cc:1261] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260924 00:23:11.548738 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:11.627671 2141 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:23:11.631626 2110 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:11.627588 2069 raft_consensus.cc:2454] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:11.634356 2140 raft_consensus.cc:2379] T 970b194b81a34f3db37adc13d58b0a01 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 10: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:11.632506 1990 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:11.635857 2643 raft_consensus.cc:2354] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:11.635078 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5e2f99029440bd9d42ec2a01200d4a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.638019 1960 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:11.645651 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:11.645560 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.646415 2068 raft_consensus.cc:1261] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260924 00:23:11.646826 2068 pending_rounds.cc:76] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting all ops after (but not including) 3
I20260924 00:23:11.647461 2068 pending_rounds.cc:97] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d: Aborting uncommitted NO_OP operation due to leader change: 7.4
W20260924 00:23:11.559875 5576 log.cc:924] Time spent T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.311s user 0.000s sys 0.003s
I20260924 00:23:11.647744 2068 raft_consensus.cc:2866] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260924 00:23:11.521891 5500 consensus_meta.cc:287] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.514s user 0.007s sys 0.004s
W20260924 00:23:11.644490 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 104185us
I20260924 00:23:11.646742 2064 raft_consensus.cc:2362] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6. Re-sending same reply.
I20260924 00:23:11.616552 5387 raft_consensus.cc:2726] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:11.620061 5581 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:11.652009 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.560515 5509 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.610687 5408 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:11.653725 2110 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:11.622891 5311 raft_consensus.cc:3037] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:11.654704 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.622373 5349 raft_consensus.cc:482] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:11.648821 5186 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:11.562600 5266 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.639308 5464 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:23:11.665904 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9aa1fbdb2cf48bb9bdb4c51026f3c83" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.668546 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.630921 5159 raft_consensus.cc:2781] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:23:11.512866 5310 raft_consensus.cc:504] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.616318 5439 raft_consensus.cc:504] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.652499 5465 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:11.629966 5456 raft_consensus.cc:504] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.676637 5458 raft_consensus.cc:2781] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:11.677102 5502 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:23:11.677640 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.678042 2035 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:11.677743 5583 consensus_queue.cc:237] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.678591 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa219f0283c4632b6601e60980ae25a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.554467 5271 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.681635 2064 raft_consensus.cc:2379] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 9: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.671941 2110 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:11.692149 5464 raft_consensus.cc:482] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:11.687675 5159 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:11.688567 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.616942 5529 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.616611 5528 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.621441 5539 raft_consensus.cc:504] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:11.620759 5547 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.618000 5143 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.670715 5392 raft_consensus.cc:3037] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:11.670969 5408 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.672873 5127 raft_consensus.cc:2781] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:23:11.672818 5349 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.727980 5159 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:11.729602 5485 raft_consensus.cc:2781] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:11.736366 5127 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.732776 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.737892 5524 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:11.757982 5524 raft_consensus.cc:482] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:11.758213 2064 raft_consensus.cc:2379] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 13: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.758409 5524 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:11.759578 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.760744 2064 raft_consensus.cc:2379] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.741657 1988 raft_consensus.cc:1226] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:23:11.745374 1993 raft_consensus.cc:1226] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260924 00:23:11.713256 5445 consensus_queue.cc:1046] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260924 00:23:11.763393 5445 consensus_queue.cc:1225] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer 9e79063532384ada8ea8f8205478adcf log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260924 00:23:11.745385 1991 raft_consensus.cc:1226] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:11.763969 5445 raft_consensus.cc:3032] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:23:11.764416 5445 raft_consensus.cc:729] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:11.766427 5342 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.768925 5271 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.770886 5456 leader_election.cc:288] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.726298 5464 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:11.714607 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:23:11.786448 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 71917.4us
I20260924 00:23:11.657284 5500 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.687107 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.719041 1995 raft_consensus.cc:1226] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [term 4 LEADER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:23:11.717751 2069 raft_consensus.cc:2454] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 2.
W20260924 00:23:11.721246 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 49389us
W20260924 00:23:11.724960 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 81330.8us
I20260924 00:23:11.665256 5282 consensus_queue.cc:1046] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260924 00:23:11.704732 5465 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:11.728694 2063 raft_consensus.cc:2379] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 10: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.730563 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f367683e243cdb37181c1bdb546ec" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.711249 5458 raft_consensus.cc:482] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
I20260924 00:23:11.713630 5502 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:11.739153 1992 raft_consensus.cc:2379] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf in current term 10: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:11.752969 2109 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:11.713950 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.804139 2062 raft_consensus.cc:2379] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 9: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:11.789265 2067 raft_consensus.cc:2362] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8. Re-sending same reply.
I20260924 00:23:11.757037 5127 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:11.748546 5485 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:11.811107 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.811933 1996 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:11.750451 5394 leader_election.cc:288] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.819948 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.810832 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.793464 5412 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:23:11.816972 2613 leader_election.cc:302] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:11.799911 5465 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:11.827808 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76aee4623d7484e8ff6c1f1a13f5fea" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.802440 5458 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:11.827963 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.805812 5502 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:11.828017 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.821127 2064 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:23:11.855764 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:11.853402 2067 raft_consensus.cc:2454] T b76aee4623d7484e8ff6c1f1a13f5fea P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:11.819032 5485 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:11.868075 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.868897 2069 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:11.871023 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.826489 5412 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:11.894133 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.894886 2067 raft_consensus.cc:3037] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:11.801755 5430 consensus_queue.cc:1046] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260924 00:23:11.902397 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.905316 2064 raft_consensus.cc:3037] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:11.922093 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.861667 5569 leader_election.cc:288] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.853554 5343 raft_consensus.cc:2781] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Leader election won for term 7
W20260924 00:23:11.934567 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 139752us
I20260924 00:23:11.932543 2066 raft_consensus.cc:2379] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:11.893632 5412 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:23:11.919179 5576 log.cc:924] Time spent T 95ac9e39e8934ecdb84ac79399dcbd9c P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260924 00:23:11.949429 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 121235us
I20260924 00:23:11.949671 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.825860 5445 consensus_queue.cc:260] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.953961 2068 raft_consensus.cc:2379] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:11.939606 5310 leader_election.cc:288] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.959501 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:11.961571 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
W20260924 00:23:11.951683 5544 consensus_meta.cc:287] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.540s user 0.010s sys 0.003s
I20260924 00:23:11.949124 5119 raft_consensus.cc:2781] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:23:11.948027 5343 raft_consensus.cc:686] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:11.975443 5290 raft_consensus.cc:3032] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:23:11.976125 5290 raft_consensus.cc:729] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
W20260924 00:23:11.976854 5591 log.cc:924] Time spent T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260924 00:23:11.981724 5343 consensus_queue.cc:237] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:11.983110 5529 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:11.953994 5439 leader_election.cc:288] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:11.986785 2071 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:12.000392 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:11.997082 2063 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:23:12.003806 2063 raft_consensus.cc:1203] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 LEADER]: Deduplicated request from leader. Original: 9.6->[11.7-11.7] Dedup: 11.7->[]
I20260924 00:23:12.004398 2063 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 LEADER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260924 00:23:12.000195 2064 raft_consensus.cc:2454] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:23:12.014169 2069 raft_consensus.cc:2454] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 11.
W20260924 00:23:12.018149 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 265222us
I20260924 00:23:11.965869 5119 raft_consensus.cc:482] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:23:12.015129 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 159469us
I20260924 00:23:12.021685 2109 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:12.023079 2067 raft_consensus.cc:2454] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 12.
I20260924 00:23:12.045688 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.046468 2067 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:12.018628 5445 raft_consensus.cc:3037] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:12.050383 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.052546 2066 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 11. Current term is 12. Ops: []
I20260924 00:23:12.061342 2064 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 10. Current term is 11. Ops: []
W20260924 00:23:12.048763 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 85140.4us
I20260924 00:23:12.062988 2035 leader_election.cc:302] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:12.066998 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 269208us
I20260924 00:23:12.070675 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.071839 2035 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:12.068940 2110 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:12.024770 5119 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:12.022393 5567 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:12.034852 5500 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.096956 2067 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
W20260924 00:23:12.078501 5589 log.cc:924] Time spent T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20260924 00:23:12.098603 1960 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:12.079389 5608 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 00:23:12.079931 5600 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20260924 00:23:12.085698 5592 log.cc:924] Time spent T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.216s user 0.001s sys 0.000s
I20260924 00:23:12.089316 2081 tablet_copy_service.cc:342] P 948356bf66a941ef8249ecbd0ec94f6d: Request end of tablet copy session 9e79063532384ada8ea8f8205478adcf-88e267ffb8c4467b93815c1a9e3d3664 received from {username='slave'} at 127.0.0.1:43194
I20260924 00:23:12.101912 1960 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:12.105772 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.102330 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:23:12.093918 5588 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20260924 00:23:12.104753 2110 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.092909 5567 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:12.110246 1960 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:12.110531 2035 leader_election.cc:398] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667) with higher term. Message: Invalid argument: T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:12.112969 2035 leader_election.cc:401] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:23:12.066695 5311 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.100368 5430 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:12.074542 5266 consensus_queue.cc:1046] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 11.7, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260924 00:23:12.101531 5608 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:12.113905 1960 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:12.103447 5500 raft_consensus.cc:2781] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election won for term 5
W20260924 00:23:12.114771 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 147389us
I20260924 00:23:12.117559 2081 tablet_copy_service.cc:433] P 948356bf66a941ef8249ecbd0ec94f6d: ending tablet copy session 9e79063532384ada8ea8f8205478adcf-88e267ffb8c4467b93815c1a9e3d3664 on tablet 88e267ffb8c4467b93815c1a9e3d3664 with peer 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.104429 5604 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:12.111275 5439 raft_consensus.cc:2781] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:23:12.123235 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.115557 5567 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:12.131637 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.134646 5605 raft_consensus.cc:2726] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260924 00:23:12.136469 5597 log.cc:924] Time spent T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260924 00:23:12.116611 5583 raft_consensus.cc:2781] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader election won for term 12
I20260924 00:23:12.116433 5430 raft_consensus.cc:482] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:12.102963 5485 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.140921 5524 raft_consensus.cc:504] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.141268 5412 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.143162 5524 leader_election.cc:288] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.143213 5412 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.144043 2062 raft_consensus.cc:2379] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 4: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:12.130110 5266 consensus_queue.cc:1225] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer 948356bf66a941ef8249ecbd0ec94f6d log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260924 00:23:12.144286 5127 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.130563 5581 raft_consensus.cc:2781] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:12.145315 2069 raft_consensus.cc:2354] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9.
I20260924 00:23:12.146435 5311 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.148468 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.138029 5500 raft_consensus.cc:686] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:12.129341 5608 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:23:12.147238 5465 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.151733 5547 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.153255 5528 leader_election.cc:288] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.111729 5408 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.181763 5465 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.182090 5408 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:12.181627 5290 consensus_queue.cc:260] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.183252 5581 raft_consensus.cc:686] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [term 10 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:12.184623 5581 consensus_queue.cc:237] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.186040 5544 raft_consensus.cc:3032] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:23:12.186391 5544 raft_consensus.cc:729] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:12.188167 2070 raft_consensus.cc:1226] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:12.189266 2069 raft_consensus.cc:1261] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260924 00:23:12.155073 5539 leader_election.cc:288] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:12.157213 5147 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:23:12.158087 5610 consensus_queue.cc:1046] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:12.190755 2070 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:12.191332 2069 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260924 00:23:12.192404 2070 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260924 00:23:12.161391 5430 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:12.195941 5240 raft_consensus.cc:3032] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Stepping down as leader of term 11
I20260924 00:23:12.129997 5604 raft_consensus.cc:482] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:12.196411 5240 raft_consensus.cc:729] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:12.162175 5611 raft_consensus.cc:3037] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:12.104602 5606 consensus_queue.cc:1046] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260924 00:23:12.163151 5533 raft_consensus.cc:3037] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:12.163450 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.196596 5604 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:12.170534 2063 raft_consensus.cc:1226] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 4. Ops: []
I20260924 00:23:12.173621 2065 raft_consensus.cc:1261] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260924 00:23:12.174840 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:23:12.176570 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 77186us
I20260924 00:23:12.177651 5271 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:12.176020 5439 raft_consensus.cc:686] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:12.177736 5349 raft_consensus.cc:504] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.178519 5583 raft_consensus.cc:686] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 12 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:12.178925 5601 consensus_queue.cc:1046] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260924 00:23:12.179149 5500 consensus_queue.cc:237] T 93bfcd566e3e4c77b3a672d5705a546f P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.188905 5159 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.153457 5458 raft_consensus.cc:504] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:12.219130 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.202203 1960 leader_election.cc:398] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9.
I20260924 00:23:12.182636 5127 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.225025 5349 leader_election.cc:288] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:12.207130 2067 raft_consensus.cc:2362] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7. Re-sending same reply.
I20260924 00:23:12.225458 5389 raft_consensus.cc:3032] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:12.170182 2071 raft_consensus.cc:1226] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:23:12.201344 5520 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:12.210971 2110 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:12.226930 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.224472 5583 consensus_queue.cc:237] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.224941 5458 leader_election.cc:288] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:12.210785 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:23:12.227782 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 113779us
I20260924 00:23:12.228752 2110 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:12.228853 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.221093 1960 leader_election.cc:401] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260924 00:23:12.211550 5614 consensus_queue.cc:1046] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260924 00:23:12.231295 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.234349 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.238112 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.238821 2070 raft_consensus.cc:2354] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9.
I20260924 00:23:12.238886 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.239686 5614 consensus_queue.cc:1225] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260924 00:23:12.241536 2646 raft_consensus.cc:2379] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:23:12.242520 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.232348 5159 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:12.248436 2068 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:12.244470 2064 raft_consensus.cc:2454] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6.
I20260924 00:23:12.247303 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.225921 5389 raft_consensus.cc:729] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:12.250628 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e92f7a727fb4c1da7807d1429d98861" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.250790 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55201d7dbea64daab55bc0144cc26c13" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.256032 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.263361 2070 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:23:12.257227 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:12.251557 2064 raft_consensus.cc:2362] T 55201d7dbea64daab55bc0144cc26c13 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 6. Re-sending same reply.
I20260924 00:23:12.234557 5349 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:23:12.266440 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:23:12.272173 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 200298us
I20260924 00:23:12.272899 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.273454 2062 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:12.274104 1996 raft_consensus.cc:1226] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:12.275676 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.278019 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f86eadee0b4bbcbfd3437d30d304b5" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.279452 2138 raft_consensus.cc:1261] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260924 00:23:12.218298 5271 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.281114 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.280355 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.284667 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.285459 2643 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:12.280798 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.240841 5528 raft_consensus.cc:3037] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:12.279853 2138 pending_rounds.cc:76] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Aborting all ops after (but not including) 6
I20260924 00:23:12.284078 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.422564 2138 pending_rounds.cc:97] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260924 00:23:12.422919 2138 raft_consensus.cc:2866] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 00:23:12.235039 5520 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.395469 1995 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:12.390108 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
W20260924 00:23:12.389874 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 121515us
I20260924 00:23:12.389626 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.240417 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.249233 5127 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260924 00:23:12.389218 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.393355 2142 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:23:12.398368 2062 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
W20260924 00:23:12.415017 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 152463us
I20260924 00:23:12.416931 2641 raft_consensus.cc:2379] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 10: Already voted for candidate 9e79063532384ada8ea8f8205478adcf in this term.
I20260924 00:23:12.418251 2646 raft_consensus.cc:3037] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:23:12.290746 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 54908us
I20260924 00:23:12.390479 5601 consensus_queue.cc:1046] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:23:12.420502 2649 raft_consensus.cc:3032] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Stepping down as leader of term 3
I20260924 00:23:12.420748 2140 raft_consensus.cc:1226] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:12.420955 1996 raft_consensus.cc:2379] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:12.393661 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1b238378448a5987c49aafc3b6313" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.262270 5266 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.131s
I20260924 00:23:12.426523 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.434965 2649 raft_consensus.cc:729] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:12.428387 1960 leader_election.cc:302] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:12.427704 2136 raft_consensus.cc:3037] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 10
I20260924 00:23:12.437997 1960 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.439438 2035 leader_election.cc:302] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:12.271611 5349 raft_consensus.cc:482] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 948356bf66a941ef8249ecbd0ec94f6d)
W20260924 00:23:12.429627 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 105456us
I20260924 00:23:12.443285 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.451251 1993 raft_consensus.cc:2379] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 in current term 5: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:12.444228 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.452507 1960 leader_election.cc:302] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:12.455291 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.455466 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:12.456568 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91be317234ca451fad76824aef0861a6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.457692 2139 raft_consensus.cc:1203] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Deduplicated request from leader. Original: 9.6->[11.7-11.7] Dedup: 11.7->[]
I20260924 00:23:12.455487 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.458633 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ac9e39e8934ecdb84ac79399dcbd9c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.472116 1995 raft_consensus.cc:3037] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:12.457999 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.457058 2137 raft_consensus.cc:1261] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260924 00:23:12.457430 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.459806 2068 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:23:12.493268 2139 raft_consensus.cc:1226] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260924 00:23:12.266754 5119 raft_consensus.cc:504] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.496608 2142 raft_consensus.cc:2379] T 6479ba76aef8450791098f4ac4fcee4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:12.498267 1960 leader_election.cc:302] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:12.457587 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d581036c5b6e448cbae056fa4fc15f78" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
W20260924 00:23:12.499810 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 60453.8us
W20260924 00:23:12.503587 5577 log.cc:924] Time spent T 47c6ea1617904bf9bf2b2d904d0e2ae5 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260924 00:23:12.504745 2138 raft_consensus.cc:2379] T 25490e5b518e4abf90527c9e0dd7723e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 8: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:12.500697 2035 leader_election.cc:302] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.458350 2649 consensus_queue.cc:260] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.507298 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.508018 1960 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.508569 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.436685 5127 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:12.513577 2649 raft_consensus.cc:3037] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
I20260924 00:23:12.449857 5547 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260924 00:23:12.513991 2137 raft_consensus.cc:1226] T 88e267ffb8c4467b93815c1a9e3d3664 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:12.443186 5349 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:12.515653 2035 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.442993 5519 raft_consensus.cc:2726] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:23:12.431911 5137 consensus_queue.cc:1046] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260924 00:23:12.393579 5389 consensus_queue.cc:260] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.457206 5520 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:12.509733 5464 consensus_meta.cc:287] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.702s user 0.009s sys 0.001s
I20260924 00:23:12.537281 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.504335 5301 raft_consensus.cc:2781] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:23:12.521358 5310 raft_consensus.cc:2726] T 88e267ffb8c4467b93815c1a9e3d3664 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:23:12.535545 2142 raft_consensus.cc:2354] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:12.510213 5485 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.545794 1995 raft_consensus.cc:2454] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:12.510514 5445 raft_consensus.cc:2781] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:12.514012 5127 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Advancing to term 10
W20260924 00:23:12.539693 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 91066.1us
I20260924 00:23:12.563299 2109 leader_election.cc:302] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.414944 5240 consensus_queue.cc:260] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.543256 2035 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.491930 5456 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260924 00:23:12.448252 5567 raft_consensus.cc:504] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
I20260924 00:23:12.569060 2110 leader_election.cc:302] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:12.524798 5394 raft_consensus.cc:2726] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:12.537868 5186 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:23:12.497411 5458 consensus_queue.cc:1046] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260924 00:23:12.541814 5581 raft_consensus.cc:482] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:12.573946 2643 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:12.521639 5342 consensus_queue.cc:1035] T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260924 00:23:12.555261 5143 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.554404 5464 raft_consensus.cc:504] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.554677 5600 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260924 00:23:12.588446 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 80497.8us
I20260924 00:23:12.574282 5465 raft_consensus.cc:2726] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 00:23:12.591135 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.555406 5607 consensus_queue.cc:1046] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
W20260924 00:23:12.659556 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 90610.2us
I20260924 00:23:12.639927 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.669596 2646 raft_consensus.cc:2454] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 4.
I20260924 00:23:12.570623 5301 raft_consensus.cc:686] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [term 8 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
W20260924 00:23:12.588510 5502 consensus_meta.cc:287] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.717s user 0.010s sys 0.000s
I20260924 00:23:12.673444 1993 raft_consensus.cc:1261] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260924 00:23:12.655644 1960 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:12.580200 5485 raft_consensus.cc:2726] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260924 00:23:12.653074 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 89845.9us
I20260924 00:23:12.509979 5439 consensus_queue.cc:237] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.678793 2143 raft_consensus.cc:1226] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 10. Current term is 11. Ops: []
I20260924 00:23:12.679139 2109 leader_election.cc:302] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.684191 1994 raft_consensus.cc:1226] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:23:12.684823 1960 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:12.686664 1960 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:12.688697 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 247898us
I20260924 00:23:12.685530 2136 raft_consensus.cc:2454] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
I20260924 00:23:12.690976 5342 consensus_queue.cc:1035] T e7be2c535b984fd6bf122044c990f72e P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260924 00:23:12.693678 5580 raft_consensus.cc:2726] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:23:12.694355 5620 raft_consensus.cc:3032] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Stepping down as leader of term 2
I20260924 00:23:12.695990 1960 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:12.704147 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.589200 5412 consensus_queue.cc:1046] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260924 00:23:12.672422 5159 raft_consensus.cc:2781] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:23:12.710312 1995 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:23:12.710975 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:12.714187 2641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.717574 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.734890 2070 raft_consensus.cc:2379] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 12: Already voted for candidate 948356bf66a941ef8249ecbd0ec94f6d in this term.
I20260924 00:23:12.736410 1987 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 12 LEADER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 11. Current term is 12. Ops: []
I20260924 00:23:12.736958 2614 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:12.734084 2646 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:23:12.688758 5485 raft_consensus.cc:2726] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260924 00:23:12.689420 5143 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:23:12.694876 5539 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260924 00:23:12.744765 2070 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260924 00:23:12.745167 2063 raft_consensus.cc:1261] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260924 00:23:12.744659 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.752907 2062 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 8.
I20260924 00:23:12.712015 5600 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:12.700762 5620 raft_consensus.cc:729] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:12.711690 5271 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:12.679018 5502 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.666074 5581 raft_consensus.cc:504] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.735335 5456 raft_consensus.cc:2726] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260924 00:23:12.771389 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:12.745668 5159 raft_consensus.cc:686] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [term 8 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:12.787412 1960 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:12.789026 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:12.757012 5282 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election won for term 8
I20260924 00:23:12.739760 5607 consensus_queue.cc:1035] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
W20260924 00:23:12.796779 5430 consensus_meta.cc:287] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.603s user 0.005s sys 0.003s
W20260924 00:23:12.797785 5623 log.cc:924] Time spent T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260924 00:23:12.801384 5282 raft_consensus.cc:686] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [term 8 LEADER]: Becoming Leader. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
W20260924 00:23:12.801657 5622 log.cc:924] Time spent T fc79ee9a960d47c2b7d3150a8aa37983 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
W20260924 00:23:12.802467 5621 log.cc:924] Time spent T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20260924 00:23:12.768450 5600 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:12.807579 5533 consensus_meta.cc:287] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.589s user 0.006s sys 0.000s
I20260924 00:23:12.809612 5567 leader_election.cc:288] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Requested vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:12.877063 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:12.878144 1996 raft_consensus.cc:2396] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260924 00:23:12.855967 5147 consensus_meta.cc:287] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.666s user 0.009s sys 0.000s
I20260924 00:23:12.762305 5119 leader_election.cc:288] T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.767294 5394 raft_consensus.cc:3032] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:12.767843 5412 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:12.815382 2649 raft_consensus.cc:1261] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260924 00:23:12.854226 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:23:12.863840 2036 connection.cc:232] RPC call timeout handler was delayed by -4.2821771950000311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.169s for a response
I20260924 00:23:12.863703 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:12.756845 5601 consensus_queue.cc:1046] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
W20260924 00:23:12.892338 2036 connection.cc:232] RPC call timeout handler was delayed by -3.8771771950000584s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21400000000000002s for a response
I20260924 00:23:12.795091 5456 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:23:12.870985 1960 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:12.910187 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 208557us
I20260924 00:23:12.911132 1960 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:12.871236 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 123586us
W20260924 00:23:12.869684 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 190602us
I20260924 00:23:12.811758 5271 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:12.875413 5282 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:12.810451 5611 consensus_meta.cc:287] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.599s user 0.004s sys 0.004s
W20260924 00:23:12.859187 5608 consensus_meta.cc:287] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.647s user 0.000s sys 0.009s
I20260924 00:23:12.891505 5464 leader_election.cc:288] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.885040 5544 consensus_queue.cc:260] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.892891 2140 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 10
I20260924 00:23:12.913605 2109 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
W20260924 00:23:13.009500 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 96041.8us
I20260924 00:23:13.009876 2138 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:23:12.854069 5430 raft_consensus.cc:504] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.894506 5147 raft_consensus.cc:2726] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 00:23:12.894146 5619 log.cc:924] Time spent T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.324s user 0.000s sys 0.001s
I20260924 00:23:12.948930 5611 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667) with higher term. Message: Invalid argument: T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:12.786180 5301 consensus_queue.cc:237] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.941917 5600 leader_election.cc:288] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:12.942281 5159 consensus_queue.cc:237] T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:12.943614 2035 leader_election.cc:302] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:12.911538 5524 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:12.916478 5143 raft_consensus.cc:2726] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260924 00:23:12.938606 5617 consensus_queue.cc:1046] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.022s
I20260924 00:23:12.937032 5539 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:23:12.887866 5394 raft_consensus.cc:729] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:12.948266 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:12.884317 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1caddf844ef49c0b3664f36bb76b869" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.044661 1990 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:12.961050 5533 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:12.961596 5629 consensus_queue.cc:1035] T 466d562826384d688b157b8830cfd5d1 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260924 00:23:12.962525 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0af13699974118a86f602c1fa9490b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:12.966372 5465 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:12.974130 1993 raft_consensus.cc:1226] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [term 11 LEADER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 10. Current term is 11. Ops: []
I20260924 00:23:13.005877 2137 raft_consensus.cc:1261] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260924 00:23:12.994935 2036 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
W20260924 00:23:12.995515 2613 leader_election.cc:334] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Timed out: RequestConsensusVote RPC to 127.0.203.2:39591 timed out after 1.898s (SENT)
I20260924 00:23:12.946480 5445 raft_consensus.cc:686] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:12.945785 5500 raft_consensus.cc:3037] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:12.914337 5271 raft_consensus.cc:482] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:13.012961 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:12.906221 5620 consensus_queue.cc:260] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:13.019345 2109 leader_election.cc:334] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Timed out: RequestConsensusVote RPC to 127.0.203.2:39591 timed out after 1.617s (SENT)
I20260924 00:23:12.974243 5608 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:12.910678 5412 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.027722 2141 raft_consensus.cc:1226] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 3. Current term is 4. Ops: []
I20260924 00:23:13.035106 2066 raft_consensus.cc:1226] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 9. Current term is 11. Ops: []
I20260924 00:23:13.034596 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.017179 5630 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260924 00:23:13.037351 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb231d6aeb14f4d83ab20c17ea2f9e1" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
I20260924 00:23:13.014154 5544 raft_consensus.cc:3037] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:12.918035 5127 raft_consensus.cc:504] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:13.045378 5634 log.cc:924] Time spent T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260924 00:23:13.045914 5430 leader_election.cc:288] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:13.121624 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 176786us
I20260924 00:23:13.046723 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.047535 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:12.956439 2142 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:13.043618 5628 consensus_queue.cc:1035] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260924 00:23:13.050434 2138 raft_consensus.cc:1226] T f63d709a34bd44a1b37eee2f1eb2112c P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 3eb374c9744b4c7e92dc749aece177dd for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:13.097385 5271 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:13.053763 2068 raft_consensus.cc:3037] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:13.059531 5465 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.060071 5640 raft_consensus.cc:482] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:13.053720 5611 raft_consensus.cc:2781] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election won for term 9
W20260924 00:23:13.064211 5604 consensus_meta.cc:287] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.837s user 0.006s sys 0.005s
I20260924 00:23:13.051450 5524 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.056636 5533 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.066844 5389 raft_consensus.cc:3037] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:13.158191 1990 raft_consensus.cc:2454] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
W20260924 00:23:13.158490 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 214955us
I20260924 00:23:13.066751 5394 consensus_queue.cc:260] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } attrs { replace: true } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: NON_VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: true } }
I20260924 00:23:13.159255 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9e32aa31c48a5b4d98f6813c8460a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:13.160272 5394 raft_consensus.cc:3037] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:13.160393 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:13.067458 5456 consensus_queue.cc:1035] T 13ff368f645e4ee7acdef88492f25ae7 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:23:13.160461 5607 consensus_queue.cc:1035] T 55201d7dbea64daab55bc0144cc26c13 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260924 00:23:13.163196 2110 leader_election.cc:302] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:23:13.070652 5349 consensus_meta.cc:287] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.516s user 0.010s sys 0.000s
I20260924 00:23:13.164616 5349 raft_consensus.cc:504] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:13.071579 5408 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260924 00:23:13.075073 2137 raft_consensus.cc:1226] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:13.054538 5502 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:13.094882 2649 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:13.115386 2140 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
W20260924 00:23:13.115602 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 171154us
I20260924 00:23:13.118057 2070 raft_consensus.cc:1261] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260924 00:23:13.119864 2145 raft_consensus.cc:2379] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 4: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:13.121807 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:13.178742 1960 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:13.013154 5606 raft_consensus.cc:482] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:13.097275 5629 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:13.135226 2136 raft_consensus.cc:3032] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Stepping down as leader of term 11
I20260924 00:23:13.135942 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447705637cdc44668a870a52a7a3871a" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:13.283031 2136 raft_consensus.cc:729] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:13.107939 5581 leader_election.cc:288] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.119860 5620 raft_consensus.cc:3037] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:23:13.141004 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ef9fa9e5dd48218a5c243de0c0e49a" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:13.161418 2035 leader_election.cc:302] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:13.129294 5392 raft_consensus.cc:3032] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:13.156620 5604 raft_consensus.cc:504] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.261444 5127 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:13.297326 5657 raft_consensus.cc:2781] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader election won for term 4
I20260924 00:23:13.297922 5657 raft_consensus.cc:686] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:13.298092 5594 raft_consensus.cc:3032] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:23:13.298529 5594 raft_consensus.cc:729] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:13.298748 5604 leader_election.cc:288] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:13.298877 5657 consensus_queue.cc:237] T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.129264 5593 consensus_queue.cc:1046] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260924 00:23:13.300179 5593 consensus_queue.cc:1225] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260924 00:23:13.300678 5593 raft_consensus.cc:3032] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Stepping down as leader of term 10
I20260924 00:23:13.300989 5593 raft_consensus.cc:729] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:13.201028 5310 raft_consensus.cc:2781] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:13.301462 5644 consensus_queue.cc:1046] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260924 00:23:13.302151 5310 raft_consensus.cc:686] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [term 10 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:13.302353 5349 leader_election.cc:288] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Requested vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:13.160360 5343 consensus_queue.cc:1046] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260924 00:23:13.233606 2036 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:13.303041 5310 consensus_queue.cc:237] T bf0af13699974118a86f602c1fa9490b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } }
W20260924 00:23:13.303840 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 70456.8us
I20260924 00:23:13.232352 5290 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Advancing to term 4
W20260924 00:23:13.235327 5651 log.cc:924] Time spent T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260924 00:23:13.306452 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2639) took 1058 ms. Trace:
I20260924 00:23:13.306766 1992 rpcz_store.cc:276] 0924 00:23:12.248025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.248139 (+ 114us) service_pool.cc:227] Handling call
0924 00:23:12.253827 (+ 5688us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:12.254045 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:12.254069 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:12.886492 (+632423us) spinlock_profiling.cc:243] Waited 8.83 ms on lock 0x7b2c000287c8. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d349fe11 00007fb6d34af442 00007fb6d4af7c11 00007fb6d4abc58a 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772
0924 00:23:12.886905 (+ 413us) log.cc:842] Serialized 26 byte log entry
0924 00:23:13.008440 (+121535us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:23:13.008489 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:13.008582 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:13.306143 (+297561us) raft_consensus.cc:1671] finished
0924 00:23:13.306160 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:13.306379 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20317440,"thread_start_us":121066,"threads_started":1,"wal-append.queue_time_us":210828}
I20260924 00:23:13.308445 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7be2c535b984fd6bf122044c990f72e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:23:13.226620 5591 log.cc:924] Time spent T 00000000000000000000000000000000 P 868c66955d2345848e02c62dafaf4a19: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260924 00:23:13.154842 5445 consensus_queue.cc:237] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:13.254170 5652 log.cc:924] Time spent T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260924 00:23:13.167912 5529 raft_consensus.cc:482] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:13.254482 5625 log.cc:924] Time spent T 2c4607ccc19a497b8082abfda6bcc0b9 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.514s user 0.004s sys 0.000s
I20260924 00:23:13.310485 2036 leader_election.cc:302] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:13.259943 5608 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 4 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:13.264824 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 53886us
I20260924 00:23:13.272186 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ff368f645e4ee7acdef88492f25ae7" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.274437 5271 raft_consensus.cc:504] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.275053 5502 consensus_queue.cc:1046] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260924 00:23:13.143157 5640 raft_consensus.cc:504] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.277566 5465 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.279064 5533 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:13.138810 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 127250us
I20260924 00:23:13.181406 5630 raft_consensus.cc:2781] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:23:13.139601 5547 consensus_queue.cc:1046] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260924 00:23:13.342832 1995 raft_consensus.cc:2354] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9.
I20260924 00:23:13.166760 5611 raft_consensus.cc:686] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
W20260924 00:23:13.295099 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.4:42667, user_credentials={real_user=slave}} blocked reactor thread for 135950us
I20260924 00:23:13.299394 5392 raft_consensus.cc:729] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:13.202997 5526 raft_consensus.cc:3032] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Stepping down as leader of term 4
I20260924 00:23:13.258656 5524 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.290249 2649 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:23:13.312355 2144 raft_consensus.cc:2354] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9.
I20260924 00:23:13.312592 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86edca4e63224252bc39bdd9c0bddca6" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.316349 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:13.319262 1960 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:13.321738 2068 raft_consensus.cc:2454] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 10.
I20260924 00:23:13.340041 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ac2f44b186407ba12e93016e1ae270" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.325830 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c2f84c42b9478e954135118d17b4e2" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:23:13.340107 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 180428us
I20260924 00:23:13.341094 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:13.351301 2070 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 13.
I20260924 00:23:13.352711 2139 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 12.
I20260924 00:23:13.358067 1992 raft_consensus.cc:2379] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 9: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
W20260924 00:23:13.357059 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 51967.5us
I20260924 00:23:13.363016 2036 leader_election.cc:302] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:13.368786 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:13.342262 5465 raft_consensus.cc:2726] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:13.372779 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:13.268924 5606 raft_consensus.cc:504] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.357240 5502 consensus_queue.cc:1225] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260924 00:23:13.282697 5629 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.349176 5526 raft_consensus.cc:729] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:13.356724 5600 raft_consensus.cc:2741] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260924 00:23:13.356694 5630 raft_consensus.cc:686] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:13.381997 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66a46f82114470b22385448fd57d51" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:13.409194 1987 raft_consensus.cc:3037] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:13.319991 5529 raft_consensus.cc:504] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.370656 5412 leader_election.cc:288] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.384709 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.364995 5524 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:13.364886 5533 raft_consensus.cc:3032] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Stepping down as leader of term 5
I20260924 00:23:13.371147 5581 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:13.321293 5342 consensus_queue.cc:1035] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260924 00:23:13.406253 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2641) took 1157 ms. Trace:
I20260924 00:23:13.387135 5430 raft_consensus.cc:2741] T 13ff368f645e4ee7acdef88492f25ae7 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260924 00:23:13.381819 5301 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260924 00:23:13.421193 2036 leader_election.cc:302] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:23:13.416109 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 89955us
I20260924 00:23:13.427199 1990 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 5.
W20260924 00:23:13.414853 5528 consensus_meta.cc:287] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd: Time spent flushing consensus metadata: real 0.981s user 0.009s sys 0.006s
I20260924 00:23:13.433920 2613 leader_election.cc:302] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:13.434139 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 74110.4us
I20260924 00:23:13.436477 5271 leader_election.cc:288] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:13.440102 5656 log.cc:924] Time spent T aa5e2f99029440bd9d42ec2a01200d4a P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
W20260924 00:23:13.440691 5649 log.cc:924] Time spent T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260924 00:23:13.468261 1987 raft_consensus.cc:2454] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:23:13.470533 2110 leader_election.cc:302] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:13.430595 5611 consensus_queue.cc:237] T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.415212 5282 consensus_queue.cc:1046] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260924 00:23:13.433485 5301 raft_consensus.cc:482] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:13.413829 5665 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260924 00:23:13.420926 1989 rpcz_store.cc:276] 0924 00:23:12.248764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.249780 (+ 1016us) service_pool.cc:227] Handling call
0924 00:23:12.269121 (+ 19341us) raft_consensus.cc:1457] Updating replica for 1 ops
0924 00:23:12.290428 (+ 21307us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:12.414508 (+124080us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0924 00:23:12.423466 (+ 8958us) log.cc:842] Serialized 26 byte log entry
0924 00:23:12.571517 (+148051us) raft_consensus.cc:1631] Marking committed up to 4
0924 00:23:12.571563 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:12.571633 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0924 00:23:13.253689 (+682056us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b44000232cc. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d19d8f11 00007fb6d1aa6a95 00007fb6d1aa74b2 00007fb6d474f511 00007fb6d474ae1a 00007fb6d473ebee 00007fb6d473df36 00007fb6d473db58 00007fb6d4747551 00007fb6d4747509 00007fb6d4747499
0924 00:23:13.385275 (+131586us) raft_consensus.cc:1671] finished
0924 00:23:13.385311 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:13.406189 (+ 20878us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2066,"raft.queue_time_us":218,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":111729,"spinlock_wait_cycles":3829504,"thread_start_us":109724,"threads_started":1,"wal-append.queue_time_us":225315}
I20260924 00:23:13.445094 5500 raft_consensus.cc:2726] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260924 00:23:13.417569 5533 raft_consensus.cc:729] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:13.442301 5640 leader_election.cc:288] T 832ee611b00d4a42b3245beb345af7ad P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.472877 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313cbaaa6f5743a1b44a28217177fb4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.506835 1987 raft_consensus.cc:2379] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 8: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:13.420051 5524 raft_consensus.cc:504] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.508376 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bfcd566e3e4c77b3a672d5705a546f" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:13.479573 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
W20260924 00:23:13.496264 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 45425.3us
I20260924 00:23:13.448673 5412 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:13.461967 2035 leader_election.cc:398] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:13.446657 5529 leader_election.cc:288] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:13.451428 5528 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for earlier term 8. Current term is 9.
I20260924 00:23:13.469213 5434 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:13.411134 5392 consensus_queue.cc:260] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.499722 5301 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 13 FOLLOWER]: Advancing to term 14
I20260924 00:23:13.497467 5604 raft_consensus.cc:2781] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election won for term 11
I20260924 00:23:13.537828 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa7fce6500d4a22b34baf31a7dbb247" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:13.555835 1987 raft_consensus.cc:3037] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:13.554359 1988 raft_consensus.cc:2454] T 447705637cdc44668a870a52a7a3871a P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:13.538465 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf"
W20260924 00:23:13.517853 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 106741us
I20260924 00:23:13.534718 2613 leader_election.cc:302] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:13.535851 1994 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 8.
I20260924 00:23:13.557278 2649 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:13.541361 2035 leader_election.cc:401] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260924 00:23:13.542448 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:13.504555 5445 raft_consensus.cc:2726] T e7be2c535b984fd6bf122044c990f72e P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260924 00:23:13.549697 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1695e8b6d44e39a87ae6431db2eb9b" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
W20260924 00:23:13.559211 2035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 109823us
I20260924 00:23:13.559434 2064 raft_consensus.cc:3037] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:13.551614 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55052 (request call id 2637) took 1304 ms. Trace:
I20260924 00:23:13.445896 5526 consensus_queue.cc:260] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.445506 5630 consensus_queue.cc:237] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.562860 2063 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260924 00:23:13.535462 5640 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:13.563565 2613 leader_election.cc:398] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9.
I20260924 00:23:13.567379 2613 leader_election.cc:401] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260924 00:23:13.563297 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bd85e8dadb4876827acd9ff4be377f" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:13.560415 1991 rpcz_store.cc:276] 0924 00:23:12.247176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.247320 (+ 144us) service_pool.cc:227] Handling call
0924 00:23:12.253431 (+ 6111us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:12.443365 (+189934us) raft_consensus.cc:1520] Early marking committed up to index 0
0924 00:23:12.443415 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0924 00:23:12.443459 (+ 44us) raft_consensus.cc:1631] Marking committed up to 7
0924 00:23:12.443492 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:13.551322 (+1107830us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b5400159854. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:13.551382 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0924 00:23:13.551548 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":419910272}
W20260924 00:23:13.570719 5648 log.cc:924] Time spent T 24240f07434f49e9bef897dac3fbe801 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.260s user 0.001s sys 0.000s
W20260924 00:23:13.571615 5661 log.cc:924] Time spent T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20260924 00:23:13.574608 2139 raft_consensus.cc:2379] T 92bd85e8dadb4876827acd9ff4be377f P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd in current term 7: Already voted for candidate 8c129c01e1c047628f6a87b001aa7258 in this term.
I20260924 00:23:13.578442 2036 leader_election.cc:302] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:13.564548 5604 raft_consensus.cc:686] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:13.582962 2648 raft_consensus.cc:2396] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3eb374c9744b4c7e92dc749aece177dd for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260924 00:23:13.566676 5629 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.632972 1994 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:13.562202 5665 raft_consensus.cc:3037] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:13.639806 1989 raft_consensus.cc:1226] T d6c2f84c42b9478e954135118d17b4e2 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9. Ops: []
W20260924 00:23:13.640026 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 66390.4us
I20260924 00:23:13.584775 5343 raft_consensus.cc:2726] T 970b194b81a34f3db37adc13d58b0a01 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260924 00:23:13.585026 5526 raft_consensus.cc:3037] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:13.650442 1994 raft_consensus.cc:1226] T bd66a46f82114470b22385448fd57d51 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260924 00:23:13.645569 1993 raft_consensus.cc:2454] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:23:13.552953 5412 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.648371 2109 leader_election.cc:302] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 948356bf66a941ef8249ecbd0ec94f6d, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:13.653494 2064 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:23:13.653561 2649 raft_consensus.cc:2454] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 9.
I20260924 00:23:13.553779 5533 consensus_queue.cc:260] T f63d709a34bd44a1b37eee2f1eb2112c P 3eb374c9744b4c7e92dc749aece177dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.656410 1993 raft_consensus.cc:1226] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:13.633601 5606 leader_election.cc:288] T cfca69df981d4ab89f1ea62a1d8c1557 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.636765 5394 raft_consensus.cc:2726] T 80ac2f44b186407ba12e93016e1ae270 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260924 00:23:13.663367 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 99887.2us
I20260924 00:23:13.662282 1987 raft_consensus.cc:2454] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 9.
I20260924 00:23:13.665378 1987 raft_consensus.cc:1226] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:13.585367 5640 raft_consensus.cc:504] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.676239 5640 leader_election.cc:288] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.649443 5607 raft_consensus.cc:2726] T 447705637cdc44668a870a52a7a3871a P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591) with higher term. Message: Invalid argument: T 447705637cdc44668a870a52a7a3871a P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9.
I20260924 00:23:13.673028 2110 leader_election.cc:302] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:13.674052 5609 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:13.679663 2613 leader_election.cc:302] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:13.668699 5524 leader_election.cc:288] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.681059 1987 raft_consensus.cc:1261] T d2ef9fa9e5dd48218a5c243de0c0e49a P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260924 00:23:13.687901 1994 raft_consensus.cc:1226] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 2. Current term is 3. Ops: []
I20260924 00:23:13.684077 1992 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:13.697458 1993 raft_consensus.cc:1226] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260924 00:23:13.698639 1993 raft_consensus.cc:1226] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260924 00:23:13.697278 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 118896us
I20260924 00:23:13.691330 5458 raft_consensus.cc:2781] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:23:13.692343 1989 raft_consensus.cc:1226] T 4c4f367683e243cdb37181c1bdb546ec P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:13.694818 2143 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 690) took 1570 ms. Trace:
I20260924 00:23:13.699718 2036 leader_election.cc:302] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:13.700376 1993 raft_consensus.cc:1226] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 7. Current term is 8. Ops: []
W20260924 00:23:13.686105 5544 consensus_meta.cc:287] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.566s user 0.009s sys 0.000s
I20260924 00:23:13.703121 2143 rpcz_store.cc:276] 0924 00:23:12.112860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.112970 (+ 110us) service_pool.cc:227] Handling call
0924 00:23:12.113290 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:12.681959 (+568669us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:13.683550 (+1001591us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b540021be54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:13.683720 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1297475840}
I20260924 00:23:13.666368 5392 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:13.702246 1987 raft_consensus.cc:1226] T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:13.702462 1995 raft_consensus.cc:1226] T 55201d7dbea64daab55bc0144cc26c13 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 5. Current term is 6. Ops: []
I20260924 00:23:13.711341 2069 raft_consensus.cc:2454] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:23:13.655103 5147 consensus_queue.cc:1046] T d6c2f84c42b9478e954135118d17b4e2 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260924 00:23:13.714389 2110 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
W20260924 00:23:13.713320 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 168722us
I20260924 00:23:13.720988 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 697) took 1465 ms. Trace:
I20260924 00:23:13.721376 2141 rpcz_store.cc:276] 0924 00:23:12.255528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.255650 (+ 122us) service_pool.cc:227] Handling call
0924 00:23:12.255968 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:13.097784 (+841816us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:13.720743 (+622959us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54003507d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:13.720924 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1774377216}
W20260924 00:23:13.727114 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 78866.9us
I20260924 00:23:13.729702 1987 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:13.730064 1960 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:13.734910 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:13.735496 2064 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:23:13.739625 1989 raft_consensus.cc:1226] T 86edca4e63224252bc39bdd9c0bddca6 P 3eb374c9744b4c7e92dc749aece177dd [term 12 LEADER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 11. Current term is 12. Ops: []
W20260924 00:23:13.741220 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 77055.4us
I20260924 00:23:13.720537 5282 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260924 00:23:13.725528 5343 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:13.725826 5544 raft_consensus.cc:3037] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:13.716014 5458 raft_consensus.cc:686] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:13.707489 5607 consensus_queue.cc:1046] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260924 00:23:13.802371 2064 raft_consensus.cc:1261] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260924 00:23:13.777392 1960 leader_election.cc:302] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd; no voters: 8c129c01e1c047628f6a87b001aa7258, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:13.738282 5583 raft_consensus.cc:2781] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:23:13.709689 5630 consensus_queue.cc:1046] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
W20260924 00:23:13.776033 5676 log.cc:924] Time spent T f2730c02ac0a4812841172254e284b90 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
W20260924 00:23:13.799257 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 99644.1us
W20260924 00:23:13.858070 5677 log.cc:924] Time spent T 91be317234ca451fad76824aef0861a6 P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260924 00:23:13.776803 5620 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
W20260924 00:23:13.842461 5633 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 3eb374c9744b4c7e92dc749aece177dd: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260924 00:23:13.827301 5629 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:13.720691 5604 consensus_queue.cc:237] T bd66a46f82114470b22385448fd57d51 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.737248 5519 consensus_queue.cc:1035] T 80ac2f44b186407ba12e93016e1ae270 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
W20260924 00:23:13.918078 2144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39164 (request call id 695) took 1651 ms (client timeout 1648 ms). Trace:
I20260924 00:23:13.922360 2036 leader_election.cc:302] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:23:13.924274 2144 rpcz_store.cc:269] 0924 00:23:12.254911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.255793 (+ 882us) service_pool.cc:227] Handling call
0924 00:23:13.850361 (+1594568us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400371254. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:13.906379 (+ 56018us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2409764608}
I20260924 00:23:13.898001 5137 raft_consensus.cc:3032] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:13.801265 5343 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:13.897699 5389 raft_consensus.cc:3032] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Stepping down as leader of term 10
I20260924 00:23:13.941988 5290 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:23:13.955825 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 222935us
I20260924 00:23:13.972240 2647 raft_consensus.cc:1226] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:13.911576 5630 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader election won for term 10
I20260924 00:23:13.922063 5539 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:13.977154 5630 raft_consensus.cc:686] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [term 10 LEADER]: Becoming Leader. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
I20260924 00:23:13.978384 5630 consensus_queue.cc:237] T 150b87da8b7e456cb68c4549975535df P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.933085 5137 raft_consensus.cc:729] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:23:13.995813 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 218501us
I20260924 00:23:13.973011 5631 raft_consensus.cc:2726] T 86edca4e63224252bc39bdd9c0bddca6 P 948356bf66a941ef8249ecbd0ec94f6d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260924 00:23:13.972193 5389 raft_consensus.cc:729] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:14.002441 1960 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:13.937186 5629 raft_consensus.cc:504] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.986035 5290 raft_consensus.cc:504] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.051844 5593 consensus_queue.cc:260] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.056514 5544 raft_consensus.cc:2726] T d6c2f84c42b9478e954135118d17b4e2 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260924 00:23:14.058193 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 51314.5us
I20260924 00:23:14.019344 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:14.002727 5687 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:14.023996 5240 raft_consensus.cc:3037] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 11 FOLLOWER]: Advancing to term 12
I20260924 00:23:14.072371 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60184 (request call id 2481) took 1619 ms. Trace:
I20260924 00:23:14.026235 5646 raft_consensus.cc:3032] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Stepping down as leader of term 6
I20260924 00:23:14.072193 5687 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.028071 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36934 (request call id 2437) took 1064 ms. Trace:
I20260924 00:23:14.073503 1988 rpcz_store.cc:276] 0924 00:23:12.963372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:13.046410 (+ 83038us) service_pool.cc:227] Handling call
0924 00:23:14.027896 (+981486us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540011c454. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:14.028010 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":992881920}
W20260924 00:23:14.030277 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 203003us
I20260924 00:23:14.074915 2036 leader_election.cc:302] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
W20260924 00:23:14.075117 2069 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43194 (request call id 789) took 1869 ms (client timeout 2094 ms). Trace:
I20260924 00:23:14.075253 5687 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
W20260924 00:23:14.075511 2069 rpcz_store.cc:269] 0924 00:23:12.205632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.207965 (+ 2333us) service_pool.cc:227] Handling call
0924 00:23:14.074941 (+1866976us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54001568d4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:14.075038 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3392405376}
I20260924 00:23:14.076074 2036 leader_election.cc:302] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258
I20260924 00:23:14.076491 5407 consensus_queue.cc:1035] T 8e92f7a727fb4c1da7807d1429d98861 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260924 00:23:14.077245 5631 raft_consensus.cc:2726] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:14.077586 2036 leader_election.cc:302] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:14.077977 5631 raft_consensus.cc:3037] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
W20260924 00:23:14.031383 5686 log.cc:924] Time spent T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260924 00:23:14.031852 5680 raft_consensus.cc:482] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
W20260924 00:23:14.033311 5681 log.cc:924] Time spent T 9700c37c6830431dbb61c9f7673b99bb P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
W20260924 00:23:14.033803 5678 log.cc:924] Time spent T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260924 00:23:14.080451 5609 raft_consensus.cc:2726] T a4917c5d11b34f7aaa5cbdcf98709f22 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:14.073304 5646 raft_consensus.cc:729] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:14.081920 5593 raft_consensus.cc:3037] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:13.999106 5412 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:14.037487 5389 consensus_queue.cc:260] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.038455 5343 leader_election.cc:288] T c616f076fa3a4e75b7d3c51e60553d74 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:14.041980 5387 raft_consensus.cc:2781] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260924 00:23:14.042080 5186 consensus_queue.cc:1035] T c616f076fa3a4e75b7d3c51e60553d74 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260924 00:23:14.042901 5539 raft_consensus.cc:2726] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260924 00:23:14.044767 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55052 (request call id 2661) took 1069 ms. Trace:
I20260924 00:23:14.084669 5389 raft_consensus.cc:3037] T 6cb231d6aeb14f4d83ab20c17ea2f9e1 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:14.087235 1990 rpcz_store.cc:276] 0924 00:23:12.975438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:13.160178 (+184740us) service_pool.cc:227] Handling call
0924 00:23:14.044619 (+884441us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b540011bf54. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:14.044713 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":588944640}
I20260924 00:23:14.082295 5646 consensus_queue.cc:260] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.046798 5301 raft_consensus.cc:504] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:13.984359 5458 consensus_queue.cc:237] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.050416 5644 raft_consensus.cc:3032] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Stepping down as leader of term 7
I20260924 00:23:14.051177 5629 leader_election.cc:288] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:14.065683 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43194 (request call id 771) took 2000 ms. Trace:
I20260924 00:23:14.136129 2066 rpcz_store.cc:276] 0924 00:23:12.065617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.156594 (+ 90977us) service_pool.cc:227] Handling call
0924 00:23:12.156940 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:13.097802 (+940862us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:14.065425 (+967623us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b54002b1454. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:14.065615 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2019207936}
I20260924 00:23:14.072795 2145 rpcz_store.cc:276] 0924 00:23:12.452461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:12.456314 (+ 3853us) service_pool.cc:227] Handling call
0924 00:23:14.072169 (+1615855us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b540037fad4. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4ac02ae 00007fb6d99fdc27 00007fb6d30e5c04 00007fb6d30e5b92 00007fb6d30e5ac1 00007fb6d30e5a3c 00007fb6d30e4cf2 00007fb6d2ee694c 00007fb6d2ee5d86 00007fb6d2ee572f
0924 00:23:14.072317 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1523686144}
I20260924 00:23:14.035481 5137 consensus_queue.cc:260] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.136821 2109 leader_election.cc:302] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:14.138262 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.086440 5387 raft_consensus.cc:482] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:14.083859 5465 raft_consensus.cc:2726] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:23:14.085500 5524 consensus_queue.cc:1046] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 00:23:14.117880 1994 raft_consensus.cc:1226] T 92bd85e8dadb4876827acd9ff4be377f P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 6. Current term is 7. Ops: []
I20260924 00:23:14.118703 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.142793 1987 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 3eb374c9744b4c7e92dc749aece177dd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:23:14.084291 2036 leader_election.cc:398] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9.
I20260924 00:23:14.143857 2036 leader_election.cc:401] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260924 00:23:14.080700 5680 raft_consensus.cc:504] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.137683 5290 leader_election.cc:288] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:14.142024 5387 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:14.100888 5544 consensus_queue.cc:1046] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260924 00:23:14.133196 5644 raft_consensus.cc:729] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9e79063532384ada8ea8f8205478adcf, State: Running, Role: LEADER
I20260924 00:23:14.146046 5646 raft_consensus.cc:3037] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:14.161386 2614 leader_election.cc:302] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
I20260924 00:23:14.172092 2109 leader_election.cc:302] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:14.134635 5609 consensus_queue.cc:1035] T 2c4607ccc19a497b8082abfda6bcc0b9 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
W20260924 00:23:14.174156 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 167699us
W20260924 00:23:14.180037 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 177924us
I20260924 00:23:14.163172 5630 raft_consensus.cc:2726] T aa5e2f99029440bd9d42ec2a01200d4a P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260924 00:23:14.191352 2109 leader_election.cc:302] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:14.197393 5631 raft_consensus.cc:2726] T 313cbaaa6f5743a1b44a28217177fb4e P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260924 00:23:14.167021 5412 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775) with higher term. Message: Invalid argument: T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 8. Current term is 9.
W20260924 00:23:14.204866 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 469000us
I20260924 00:23:14.206506 2109 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:14.167590 5137 raft_consensus.cc:3037] T f2f86eadee0b4bbcbfd3437d30d304b5 P 8c129c01e1c047628f6a87b001aa7258 [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:14.168004 5605 raft_consensus.cc:3032] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 LEADER]: Stepping down as leader of term 10
I20260924 00:23:14.151585 5689 consensus_queue.cc:1046] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
W20260924 00:23:14.214711 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 130532us
I20260924 00:23:14.160526 5583 raft_consensus.cc:686] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [term 9 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
I20260924 00:23:14.215502 1987 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260924 00:23:14.180018 5465 raft_consensus.cc:3032] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Stepping down as leader of term 9
I20260924 00:23:14.217989 2109 leader_election.cc:302] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:14.225832 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:14.225808 5465 raft_consensus.cc:729] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:14.229807 1988 raft_consensus.cc:1226] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Rejecting Update request from peer 8c129c01e1c047628f6a87b001aa7258 for earlier term 11. Current term is 12. Ops: []
I20260924 00:23:14.230702 5290 consensus_queue.cc:1046] T 95ac9e39e8934ecdb84ac79399dcbd9c P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:23:14.230578 5465 consensus_queue.cc:260] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.231938 5465 raft_consensus.cc:3037] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Advancing to term 10
I20260924 00:23:14.232017 1995 raft_consensus.cc:1261] T f2730c02ac0a4812841172254e284b90 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Refusing update from remote peer 8c129c01e1c047628f6a87b001aa7258: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260924 00:23:14.226629 2063 raft_consensus.cc:2454] T 771618125e684526a907a103e7694ae8 P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:23:14.234133 5301 leader_election.cc:288] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 14 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:14.235536 2613 leader_election.cc:398] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9.
W20260924 00:23:14.238828 2613 leader_election.cc:334] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591): Timed out: RequestConsensusVote RPC to 127.0.203.2:39591 timed out after 2.072s (SENT)
I20260924 00:23:14.239264 2613 leader_election.cc:302] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e79063532384ada8ea8f8205478adcf; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:14.231086 5693 raft_consensus.cc:2726] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:14.207402 5526 raft_consensus.cc:2781] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 00:23:14.202306 5407 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.251763 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea92f011f94575ab225f58d2a0db06" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:14.256036 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.255190 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.214756 5605 raft_consensus.cc:729] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:14.258389 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b87da8b7e456cb68c4549975535df" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.292834 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832ee611b00d4a42b3245beb345af7ad" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.293921 1990 raft_consensus.cc:2454] T 150b87da8b7e456cb68c4549975535df P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 10.
I20260924 00:23:14.258854 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfca69df981d4ab89f1ea62a1d8c1557" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.289652 1992 raft_consensus.cc:1226] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 7. Current term is 8. Ops: []
I20260924 00:23:14.259914 1960 leader_election.cc:302] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:14.284636 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd"
I20260924 00:23:14.241438 5630 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Advancing to term 13
I20260924 00:23:14.318614 1988 raft_consensus.cc:3037] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Advancing to term 9
I20260924 00:23:14.304972 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36934 (request call id 2443) took 1339 ms. Trace:
I20260924 00:23:14.327462 1992 rpcz_store.cc:276] 0924 00:23:12.965776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:13.697859 (+732083us) service_pool.cc:227] Handling call
0924 00:23:14.289359 (+591500us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b5400074328. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d99637f1 00007fb6d99de2dc 00007fb6d9a1237c 00007fb6d99fd80b 00007fb6d99fd2a2 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c
0924 00:23:14.289491 (+ 132us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:14.304655 (+ 15164us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:14.304905 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10267904}
I20260924 00:23:14.330034 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.330837 2063 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 13 FOLLOWER]: Advancing to term 14
I20260924 00:23:14.296809 1987 raft_consensus.cc:2379] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 14: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:14.283089 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "466d562826384d688b157b8830cfd5d1" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.338719 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.187361 5690 consensus_queue.cc:1035] T 771618125e684526a907a103e7694ae8 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260924 00:23:14.340214 1992 raft_consensus.cc:2396] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260924 00:23:14.323513 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.323917 2145 raft_consensus.cc:2454] T aaea92f011f94575ab225f58d2a0db06 P 8c129c01e1c047628f6a87b001aa7258 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 4.
I20260924 00:23:14.228477 5392 raft_consensus.cc:2726] T 13ff368f645e4ee7acdef88492f25ae7 P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:14.346725 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:14.289304 5544 raft_consensus.cc:482] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.259627 5430 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:14.253964 5526 raft_consensus.cc:482] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 00:23:14.271884 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 54132.4us
W20260924 00:23:14.349977 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 90105.7us
I20260924 00:23:14.260218 5524 consensus_queue.cc:1046] T f2730c02ac0a4812841172254e284b90 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20260924 00:23:14.342244 1993 raft_consensus.cc:1261] T 85bede790e49439d97e8031810eee902 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Refusing update from remote peer 948356bf66a941ef8249ecbd0ec94f6d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260924 00:23:14.347378 1989 raft_consensus.cc:2454] T 6479ba76aef8450791098f4ac4fcee4e P 3eb374c9744b4c7e92dc749aece177dd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 7.
I20260924 00:23:14.284627 5687 raft_consensus.cc:3032] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Stepping down as leader of term 11
I20260924 00:23:14.284338 5689 raft_consensus.cc:2781] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:23:14.249321 5412 consensus_queue.cc:1035] T d2ef9fa9e5dd48218a5c243de0c0e49a P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20260924 00:23:14.324852 5407 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.358951 5581 raft_consensus.cc:2781] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260924 00:23:14.403852 1988 raft_consensus.cc:2454] T 25490e5b518e4abf90527c9e0dd7723e P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948356bf66a941ef8249ecbd0ec94f6d in term 9.
I20260924 00:23:14.384974 2063 raft_consensus.cc:2454] T c4982c422b5247e29807c869b18b7eb8 P 948356bf66a941ef8249ecbd0ec94f6d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 14.
W20260924 00:23:14.417119 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 54092.9us
I20260924 00:23:14.424973 1960 leader_election.cc:302] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
W20260924 00:23:14.417114 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 174303us
I20260924 00:23:14.255262 5644 consensus_queue.cc:260] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.421288 5581 raft_consensus.cc:482] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:14.362763 5683 raft_consensus.cc:482] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.370785 5689 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:14.499332 5636 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.207s
I20260924 00:23:14.286026 5594 consensus_queue.cc:260] T 93bfcd566e3e4c77b3a672d5705a546f P 9e79063532384ada8ea8f8205478adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.501267 5696 raft_consensus.cc:2726] T 4a1695e8b6d44e39a87ae6431db2eb9b P 9e79063532384ada8ea8f8205478adcf [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260924 00:23:14.378139 5665 consensus_meta.cc:287] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.731s user 0.007s sys 0.004s
I20260924 00:23:14.502013 5665 raft_consensus.cc:2726] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385) with higher term. Message: Invalid argument: T e4a9343576d1478caf59e9f5629ce211 P 8c129c01e1c047628f6a87b001aa7258 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for earlier term 10. Current term is 11.
I20260924 00:23:14.481587 5680 leader_election.cc:288] T 2f75d53c3ccf437cac00ba5d83b2c165 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:14.482146 5387 raft_consensus.cc:504] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.292506 5583 consensus_queue.cc:237] T 4a1695e8b6d44e39a87ae6431db2eb9b P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.503353 1994 raft_consensus.cc:2379] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d in current term 6: Already voted for candidate 3eb374c9744b4c7e92dc749aece177dd in this term.
I20260924 00:23:14.504202 5387 leader_election.cc:288] T 2a03e76a8d83458ca79bbaef1eee19cb P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 election: Requested vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:14.483494 5412 raft_consensus.cc:482] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
W20260924 00:23:14.483664 2109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 59775.6us
I20260924 00:23:14.510543 2109 leader_election.cc:302] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:14.364030 5430 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.364470 5526 raft_consensus.cc:3037] T aaea92f011f94575ab225f58d2a0db06 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Advancing to term 5
I20260924 00:23:14.370617 5687 raft_consensus.cc:729] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 8c129c01e1c047628f6a87b001aa7258, State: Running, Role: LEADER
W20260924 00:23:14.495622 1960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 70776.4us
I20260924 00:23:14.498901 5644 raft_consensus.cc:3037] T a4917c5d11b34f7aaa5cbdcf98709f22 P 9e79063532384ada8ea8f8205478adcf [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:14.496735 5700 raft_consensus.cc:2781] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 14 FOLLOWER]: Leader election won for term 14
I20260924 00:23:14.499718 5689 raft_consensus.cc:3037] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 2 FOLLOWER]: Advancing to term 3
W20260924 00:23:14.480815 5434 consensus_meta.cc:287] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf: Time spent flushing consensus metadata: real 0.927s user 0.010s sys 0.001s
I20260924 00:23:14.370982 5629 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.500921 5240 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.498216 5581 raft_consensus.cc:3037] T 771618125e684526a907a103e7694ae8 P 3eb374c9744b4c7e92dc749aece177dd [term 6 FOLLOWER]: Advancing to term 7
I20260924 00:23:14.501026 5143 raft_consensus.cc:482] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.522644 2109 leader_election.cc:302] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258, 948356bf66a941ef8249ecbd0ec94f6d; no voters: 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:14.363437 5544 raft_consensus.cc:504] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.528880 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708537d593c14817accda67f9fd081b4" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:14.530215 2064 raft_consensus.cc:1261] T 8e92f7a727fb4c1da7807d1429d98861 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Refusing update from remote peer 9e79063532384ada8ea8f8205478adcf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260924 00:23:14.537026 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c616f076fa3a4e75b7d3c51e60553d74" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:14.537539 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24240f07434f49e9bef897dac3fbe801" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.536481 2063 raft_consensus.cc:2454] T 708537d593c14817accda67f9fd081b4 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 7.
I20260924 00:23:14.538110 2064 raft_consensus.cc:2362] T 24240f07434f49e9bef897dac3fbe801 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8. Re-sending same reply.
I20260924 00:23:14.538021 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.554209 2071 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:14.554512 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:14.537669 2068 raft_consensus.cc:2454] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 8.
I20260924 00:23:14.560065 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479ba76aef8450791098f4ac4fcee4e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.560730 2068 raft_consensus.cc:2362] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7. Re-sending same reply.
I20260924 00:23:14.415011 5605 consensus_queue.cc:260] T bd66a46f82114470b22385448fd57d51 P 948356bf66a941ef8249ecbd0ec94f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.562973 2067 raft_consensus.cc:1226] T 93bfcd566e3e4c77b3a672d5705a546f P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 4. Current term is 5. Ops: []
I20260924 00:23:14.561916 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:14.563627 2145 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 5.
I20260924 00:23:14.524072 5434 raft_consensus.cc:2726] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260924 00:23:14.524016 5690 raft_consensus.cc:2726] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:14.565068 2064 raft_consensus.cc:2454] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 10.
I20260924 00:23:14.572825 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba683df200644d888896fb84dbc748e8" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:14.526572 5700 raft_consensus.cc:686] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [term 14 LEADER]: Becoming Leader. State: Replica: 3eb374c9744b4c7e92dc749aece177dd, State: Running, Role: LEADER
W20260924 00:23:14.567613 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 81317.3us
I20260924 00:23:14.570420 2109 leader_election.cc:302] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
I20260924 00:23:14.594405 2109 leader_election.cc:302] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c129c01e1c047628f6a87b001aa7258; no voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d
W20260924 00:23:14.654350 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.3:36385, user_credentials={real_user=slave}} blocked reactor thread for 60698.2us
I20260924 00:23:14.654593 2144 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258 [term 9 LEADER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9. Ops: []
I20260924 00:23:14.515384 5412 raft_consensus.cc:504] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.655301 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771618125e684526a907a103e7694ae8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258" is_pre_election: true
I20260924 00:23:14.663328 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700c37c6830431dbb61c9f7673b99bb" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.662653 2145 raft_consensus.cc:1261] T bf0af13699974118a86f602c1fa9490b P 8c129c01e1c047628f6a87b001aa7258 [term 10 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260924 00:23:14.561048 5137 raft_consensus.cc:2781] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260924 00:23:14.683552 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4607ccc19a497b8082abfda6bcc0b9" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.557466 5713 raft_consensus.cc:482] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:14.684834 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970b194b81a34f3db37adc13d58b0a01" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.714368 2067 raft_consensus.cc:2354] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 10. Current term is 11.
I20260924 00:23:14.509739 5683 raft_consensus.cc:504] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:14.717831 2613 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.2:39591, user_credentials={real_user=slave}} blocked reactor thread for 62066.2us
I20260924 00:23:14.712483 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4982c422b5247e29807c869b18b7eb8" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:14.526713 5629 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.722749 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25490e5b518e4abf90527c9e0dd7723e" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.725322 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a03e76a8d83458ca79bbaef1eee19cb" candidate_uuid: "3eb374c9744b4c7e92dc749aece177dd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "8c129c01e1c047628f6a87b001aa7258"
I20260924 00:23:14.572173 5693 raft_consensus.cc:482] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:14.535744 5240 raft_consensus.cc:504] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.738360 2141 raft_consensus.cc:3037] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:14.738466 2063 raft_consensus.cc:2354] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c129c01e1c047628f6a87b001aa7258 for earlier term 8. Current term is 9.
I20260924 00:23:14.740487 2110 leader_election.cc:302] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 8c129c01e1c047628f6a87b001aa7258; no voters:
I20260924 00:23:14.554678 5143 raft_consensus.cc:504] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:14.769623 2036 leader_election.cc:334] T d1caddf844ef49c0b3664f36bb76b869 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775): Timed out: RequestConsensusVote RPC to 127.0.203.1:41775 timed out after 1.955s (SENT)
I20260924 00:23:14.683051 5524 raft_consensus.cc:482] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.778622 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e267ffb8c4467b93815c1a9e3d3664" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.683398 5458 raft_consensus.cc:2726] T d2ef9fa9e5dd48218a5c243de0c0e49a P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260924 00:23:14.711814 5690 raft_consensus.cc:2726] T d1caddf844ef49c0b3664f36bb76b869 P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260924 00:23:14.710371 5711 raft_consensus.cc:482] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.718214 2644 raft_consensus.cc:1226] T 54b1b238378448a5987c49aafc3b6313 P 9e79063532384ada8ea8f8205478adcf [term 10 FOLLOWER]: Rejecting Update request from peer 948356bf66a941ef8249ecbd0ec94f6d for earlier term 9. Current term is 10. Ops: []
I20260924 00:23:14.831668 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9343576d1478caf59e9f5629ce211" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.822381 5670 raft_consensus.cc:482] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.710330 5137 raft_consensus.cc:482] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:14.662635 5691 consensus_queue.cc:1035] T fc79ee9a960d47c2b7d3150a8aa37983 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.162s
I20260924 00:23:14.861909 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2730c02ac0a4812841172254e284b90" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.868425 2141 raft_consensus.cc:2454] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb374c9744b4c7e92dc749aece177dd in term 6.
I20260924 00:23:14.880347 1996 raft_consensus.cc:2396] T d1caddf844ef49c0b3664f36bb76b869 P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948356bf66a941ef8249ecbd0ec94f6d for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260924 00:23:14.900501 2068 raft_consensus.cc:2454] T 2c4607ccc19a497b8082abfda6bcc0b9 P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 7.
I20260924 00:23:14.936398 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bede790e49439d97e8031810eee902" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d" is_pre_election: true
I20260924 00:23:14.905556 2645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f75d53c3ccf437cac00ba5d83b2c165" candidate_uuid: "948356bf66a941ef8249ecbd0ec94f6d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9e79063532384ada8ea8f8205478adcf" is_pre_election: true
I20260924 00:23:14.958436 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d709a34bd44a1b37eee2f1eb2112c" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:14.880518 5690 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.880913 5137 raft_consensus.cc:3037] T b76aee4623d7484e8ff6c1f1a13f5fea P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Advancing to term 9
W20260924 00:23:14.977277 2036 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 252695us
I20260924 00:23:14.664155 5687 consensus_queue.cc:260] T 20c9e32aa31c48a5b4d98f6813c8460a P 8c129c01e1c047628f6a87b001aa7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.825016 5693 raft_consensus.cc:504] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:14.891402 5660 raft_consensus.cc:482] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.727223 5713 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.957674 5723 raft_consensus.cc:482] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:14.851130 5524 raft_consensus.cc:504] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:15.002440 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc79ee9a960d47c2b7d3150a8aa37983" candidate_uuid: "8c129c01e1c047628f6a87b001aa7258" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "948356bf66a941ef8249ecbd0ec94f6d"
I20260924 00:23:15.017937 1989 raft_consensus.cc:1226] T efa7fce6500d4a22b34baf31a7dbb247 P 3eb374c9744b4c7e92dc749aece177dd [term 9 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260924 00:23:14.978856 5696 raft_consensus.cc:482] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:14.952131 5706 consensus_queue.cc:1046] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.108s
I20260924 00:23:15.033452 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66641de0607c476a958313ea66dff4db" candidate_uuid: "9e79063532384ada8ea8f8205478adcf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3eb374c9744b4c7e92dc749aece177dd" is_pre_election: true
I20260924 00:23:15.034147 1988 raft_consensus.cc:2454] T 66641de0607c476a958313ea66dff4db P 3eb374c9744b4c7e92dc749aece177dd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e79063532384ada8ea8f8205478adcf in term 3.
I20260924 00:23:15.023881 5430 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:15.037655 2063 raft_consensus.cc:3037] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 5 FOLLOWER]: Advancing to term 6
I20260924 00:23:15.019585 5544 leader_election.cc:288] T 9700c37c6830431dbb61c9f7673b99bb P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:15.014019 5407 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:14.886535 5670 raft_consensus.cc:504] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
W20260924 00:23:15.054975 5716 log.cc:924] Time spent T 8e92f7a727fb4c1da7807d1429d98861 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20260924 00:23:15.055862 5407 raft_consensus.cc:482] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:15.056039 5670 leader_election.cc:288] T f2f86eadee0b4bbcbfd3437d30d304b5 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:15.056393 5407 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 9e79063532384ada8ea8f8205478adcf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:15.058127 5593 consensus_meta.cc:287] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf: Time spent flushing consensus metadata: real 0.976s user 0.008s sys 0.003s
I20260924 00:23:15.058929 2646 raft_consensus.cc:1261] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Refusing update from remote peer 3eb374c9744b4c7e92dc749aece177dd: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260924 00:23:15.060642 5718 raft_consensus.cc:482] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:15.061262 5718 raft_consensus.cc:504] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:15.063386 5718 leader_election.cc:288] T 66641de0607c476a958313ea66dff4db P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:15.069283 5240 leader_election.cc:288] T 313cbaaa6f5743a1b44a28217177fb4e P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:13.536260 812 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260924 00:23:15.072499 5714 raft_consensus.cc:2741] T 2a03e76a8d83458ca79bbaef1eee19cb P 8c129c01e1c047628f6a87b001aa7258 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260924 00:23:15.073822 2144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39164 (request call id 707) took 1147 ms. Trace:
I20260924 00:23:15.083600 2067 raft_consensus.cc:2454] T 88e267ffb8c4467b93815c1a9e3d3664 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:15.083534 2144 rpcz_store.cc:276] 0924 00:23:13.926259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0924 00:23:13.931706 (+ 5447us) service_pool.cc:227] Handling call
0924 00:23:13.932074 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0924 00:23:14.662229 (+730155us) raft_consensus.cc:1683] Filling consensus response to leader.
0924 00:23:15.073546 (+411317us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b5400371254. stack: 00007fb6d1b62033 00007fb6d1b61f30 00007fb6d14c0534 00007fb6db1cc8b4 00007fb6db1cc849 00007fb6d4a41471 00007fb6d4abcf51 00007fb6d4abb99e 00007fb6d99fd2e9 00007fb6d30e3684 00007fb6d30e3612 00007fb6d30e3541 00007fb6d30e34bc 00007fb6d30e2772 00007fb6d2ee694c 00007fb6d2ee5d86
0924 00:23:15.073757 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1661245184}
W20260924 00:23:15.086391 5719 log.cc:924] Time spent T efa7fce6500d4a22b34baf31a7dbb247 P 8c129c01e1c047628f6a87b001aa7258: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260924 00:23:15.087785 5430 raft_consensus.cc:482] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:15.088310 5430 raft_consensus.cc:504] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:15.051657 5387 raft_consensus.cc:482] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:15.090276 5430 leader_election.cc:288] T 970b194b81a34f3db37adc13d58b0a01 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:15.090040 5387 raft_consensus.cc:504] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:15.091898 5387 leader_election.cc:288] T 20c9e32aa31c48a5b4d98f6813c8460a P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:15.091938 5430 raft_consensus.cc:482] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3eb374c9744b4c7e92dc749aece177dd)
I20260924 00:23:15.092504 5430 raft_consensus.cc:504] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:15.029757 5696 raft_consensus.cc:504] T 466d562826384d688b157b8830cfd5d1 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:14.891086 5711 raft_consensus.cc:504] T c616f076fa3a4e75b7d3c51e60553d74 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:15.040787 5706 consensus_queue.cc:1225] T 86edca4e63224252bc39bdd9c0bddca6 P 8c129c01e1c047628f6a87b001aa7258 [LEADER]: Peer 3eb374c9744b4c7e92dc749aece177dd log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260924 00:23:15.101655 2068 raft_consensus.cc:2454] T fc79ee9a960d47c2b7d3150a8aa37983 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 8.
I20260924 00:23:14.993731 5660 raft_consensus.cc:504] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:15.103608 2036 leader_election.cc:302] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:15.124724 2063 raft_consensus.cc:2454] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 6.
I20260924 00:23:15.096851 2065 raft_consensus.cc:2454] T e4a9343576d1478caf59e9f5629ce211 P 948356bf66a941ef8249ecbd0ec94f6d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c129c01e1c047628f6a87b001aa7258 in term 11.
I20260924 00:23:14.981662 5690 raft_consensus.cc:504] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:15.073125 5465 consensus_meta.cc:287] T 54b1b238378448a5987c49aafc3b6313 P 948356bf66a941ef8249ecbd0ec94f6d: Time spent flushing consensus metadata: real 0.841s user 0.008s sys 0.004s
I20260924 00:23:15.137272 2036 leader_election.cc:302] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948356bf66a941ef8249ecbd0ec94f6d; no voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf
I20260924 00:23:15.139369 2036 leader_election.cc:302] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:15.140730 2036 leader_election.cc:302] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 948356bf66a941ef8249ecbd0ec94f6d; no voters:
I20260924 00:23:15.146838 5524 leader_election.cc:288] T f63d709a34bd44a1b37eee2f1eb2112c P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:15.147363 5524 raft_consensus.cc:2726] T 466d562826384d688b157b8830cfd5d1 P 948356bf66a941ef8249ecbd0ec94f6d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 00:23:15.148026 5524 raft_consensus.cc:2781] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260924 00:23:15.148320 5524 raft_consensus.cc:482] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:15.148629 5524 raft_consensus.cc:3037] T 6479ba76aef8450791098f4ac4fcee4e P 948356bf66a941ef8249ecbd0ec94f6d [term 7 FOLLOWER]: Advancing to term 8
I20260924 00:23:15.149227 5718 consensus_queue.cc:1035] T 85bede790e49439d97e8031810eee902 P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:23:15.149904 5713 leader_election.cc:288] T 7fa219f0283c4632b6601e60980ae25a P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:15.150410 5713 raft_consensus.cc:2781] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 FOLLOWER]: Leader election won for term 9
I20260924 00:23:14.957975 5700 consensus_queue.cc:237] T c4982c422b5247e29807c869b18b7eb8 P 3eb374c9744b4c7e92dc749aece177dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:15.150857 5713 raft_consensus.cc:686] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [term 9 LEADER]: Becoming Leader. State: Replica: 948356bf66a941ef8249ecbd0ec94f6d, State: Running, Role: LEADER
I20260924 00:23:15.152601 5544 raft_consensus.cc:2781] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260924 00:23:15.153016 5544 raft_consensus.cc:482] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:15.153359 5544 raft_consensus.cc:3037] T 150b87da8b7e456cb68c4549975535df P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Advancing to term 11
I20260924 00:23:15.078205 5412 leader_election.cc:288] T 4c4f367683e243cdb37181c1bdb546ec P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
W20260924 00:23:15.155795 5689 consensus_meta.cc:287] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.618s user 0.008s sys 0.000s
I20260924 00:23:15.156183 5689 raft_consensus.cc:504] T 7fa219f0283c4632b6601e60980ae25a P 8c129c01e1c047628f6a87b001aa7258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:15.156901 5690 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:15.157790 5629 leader_election.cc:288] T d581036c5b6e448cbae056fa4fc15f78 P 3eb374c9744b4c7e92dc749aece177dd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:15.078230 5620 raft_consensus.cc:482] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:15.158576 5660 leader_election.cc:288] T 91be317234ca451fad76824aef0861a6 P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:15.158473 5620 raft_consensus.cc:504] T 80ac2f44b186407ba12e93016e1ae270 P 9e79063532384ada8ea8f8205478adcf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
I20260924 00:23:15.159883 5693 leader_election.cc:288] T e7be2c535b984fd6bf122044c990f72e P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 948356bf66a941ef8249ecbd0ec94f6d (127.0.203.2:39591)
I20260924 00:23:15.160398 5683 leader_election.cc:288] T 95ac9e39e8934ecdb84ac79399dcbd9c P 8c129c01e1c047628f6a87b001aa7258 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 9e79063532384ada8ea8f8205478adcf (127.0.203.4:42667)
I20260924 00:23:15.161257 5430 leader_election.cc:288] T f9aa1fbdb2cf48bb9bdb4c51026f3c83 P 948356bf66a941ef8249ecbd0ec94f6d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3eb374c9744b4c7e92dc749aece177dd (127.0.203.1:41775), 8c129c01e1c047628f6a87b001aa7258 (127.0.203.3:36385)
I20260924 00:23:15.025112 5723 raft_consensus.cc:504] T 54b1b238378448a5987c49aafc3b6313 P 3eb374c9744b4c7e92dc749aece177dd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "9e79063532384ada8ea8f8205478adcf" member_type: VOTER last_known_addr { host: "127.0.203.4" port: 42667 } attrs { promote: false } }
W20260924 00:23:15.102721 5646 consensus_meta.cc:287] T 92bd85e8dadb4876827acd9ff4be377f P 9e79063532384ada8ea8f8205478adcf: Time spent flushing consensus metadata: real 0.929s user 0.010s sys 0.000s
I20260924 00:23:15.075428 2644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45594 (request call id 678) took 1158 ms. Trace:
I20260924 00:23:15.183456 2067 raft_consensus.cc:1226] T a4917c5d11b34f7aaa5cbdcf98709f22 P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Rejecting Update request from peer 9e79063532384ada8ea8f8205478adcf for earlier term 7. Current term is 8. Ops: []
W20260924 00:23:15.164382 5630 consensus_meta.cc:287] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258: Time spent flushing consensus metadata: real 0.825s user 0.006s sys 0.001s
I20260924 00:23:15.190701 5630 raft_consensus.cc:2726] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260924 00:23:15.077127 5636 consensus_queue.cc:1046] T efa7fce6500d4a22b34baf31a7dbb247 P 9e79063532384ada8ea8f8205478adcf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.106s
I20260924 00:23:15.228631 2143 raft_consensus.cc:3037] T c4982c422b5247e29807c869b18b7eb8 P 8c129c01e1c047628f6a87b001aa7258 [term 13 FOLLOWER]: Advancing to term 14
I20260924 00:23:15.150617 5718 raft_consensus.cc:482] T 9ff940e15e1849efb194c908520c212f P 948356bf66a941ef8249ecbd0ec94f6d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9e79063532384ada8ea8f8205478adcf)
I20260924 00:23:15.151647 5713 consensus_queue.cc:237] T 25490e5b518e4abf90527c9e0dd7723e P 948356bf66a941ef8249ecbd0ec94f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb374c9744b4c7e92dc749aece177dd" member_type: VOTER last_known_addr { host: "127.0.203.1" port: 41775 } } peers { permanent_uuid: "948356bf66a941ef8249ecbd0ec94f6d" member_type: VOTER last_known_addr { host: "127.0.203.2" port: 39591 } } peers { permanent_uuid: "8c129c01e1c047628f6a87b001aa7258" member_type: VOTER last_known_addr { host: "127.0.203.3" port: 36385 } }
I20260924 00:23:15.179441 5240 raft_consensus.cc:482] T 24240f07434f49e9bef897dac3fbe801 P 3eb374c9744b4c7e92dc749aece177dd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 8c129c01e1c047628f6a87b001aa7258)
I20260924 00:23:15.188755 5465 raft_consensus.cc:482] T bf0af13699974118a86f602c1fa9490b P 948356bf66a941ef8249ecbd0ec94f6d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 00:23:15.241153 2614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.203.1:41775, user_credentials={real_user=slave}} blocked reactor thread for 205497us
I20260924 00:23:15.224773 5683 raft_consensus.cc:482] T 313cbaaa6f5743a1b44a28217177fb4e P 8c129c01e1c047628f6a87b001aa7258 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:15.243983 2614 leader_election.cc:302] T 66641de0607c476a958313ea66dff4db P 9e79063532384ada8ea8f8205478adcf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb374c9744b4c7e92dc749aece177dd, 9e79063532384ada8ea8f8205478adcf; no voters:
W20260924 00:23:15.249254 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote ->